Disable prompting for passphrase while (re)loading SSL config files.

OpenSSL's default behavior when loading a passphrase-protected key file
is to open /dev/tty and demand the password from there.  It was kinda
sorta okay to allow that to happen at server start, but really that was
never workable in standard daemon environments.  And it was a complete
fail on Windows, where the same thing would happen at every backend launch.
Yesterday's commit de41869b put the final nail in the coffin by causing
that to happen at every SIGHUP; even if you've still got a terminal acting
as the server's TTY, having the postmaster freeze until you enter the
passphrase again isn't acceptable.

Hence, override the default behavior with a callback that returns an empty
string, ensuring failure.  Change the documentation to say that you can't
have a passphrase-protected server key, period.

If we can think of a production-grade way of collecting a passphrase from
somewhere, we might do that once at server startup and use this callback
to feed it to OpenSSL, but it's far from clear that anyone cares enough
to invest that much work in the feature.  The lack of complaints about
the existing fractionally-baked behavior suggests nobody's using it anyway.

Discussion: https://postgr.es/m/29982.1483412575@sss.pgh.pa.us

doc/src/sgml/runtime.sgml

@@ -2159,9 +2159,8 @@ pg_dumpall -p 5432 | psql -d postgres -p 5433
   </para>
 
   <para>
-   If the private key is protected with a passphrase, the
-   server will prompt for the passphrase and will not start until it has
-   been entered.
+   The private key cannot be protected with a passphrase, as there is no
+   way to supply the passphrase to the server.
   </para>
 
   <para>
@@ -2315,8 +2314,8 @@ openssl req -new -text -out server.req
     you enter the local host name as <quote>Common Name</>; the challenge
     password can be left blank. The program will generate a key that is
     passphrase protected; it will not accept a passphrase that is less
-    than four characters long. To remove the passphrase (as you must if
-    you want automatic start-up of the server), run the commands:
+    than four characters long.  To remove the passphrase again (as you must),
+    next run the commands:
 <programlisting>
 openssl rsa -in privkey.pem -out server.key
 rm privkey.pem

src/backend/libpq/be-secure-openssl.c

@@ -75,6 +75,7 @@ static DH  *load_dh_file(int keylength);
 static DH  *load_dh_buffer(const char *, size_t);
 static DH  *generate_dh_parameters(int prime_len, int generator);
 static DH  *tmp_dh_cb(SSL *s, int is_export, int keylength);
+static int	ssl_passwd_cb(char *buf, int size, int rwflag, void *userdata);
 static int	verify_cb(int, X509_STORE_CTX *);
 static void info_cb(const SSL *ssl, int type, int args);
 static bool initialize_ecdh(SSL_CTX *context, bool failOnError);
@@ -203,6 +204,11 @@ be_tls_init(bool failOnError)
 	 */
 	SSL_CTX_set_mode(context, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER);
 
+	/*
+	 * Override OpenSSL's default handling of passphrase-protected files.
+	 */
+	SSL_CTX_set_default_passwd_cb(context, ssl_passwd_cb);
+
 	/*
 	 * Load and verify server's certificate and private key
 	 */
@@ -1060,6 +1066,29 @@ tmp_dh_cb(SSL *s, int is_export, int keylength)
 	return r;
 }
 
+/*
+ *	Passphrase collection callback
+ *
+ * If OpenSSL is told to use a passphrase-protected server key, by default
+ * it will issue a prompt on /dev/tty and try to read a key from there.
+ * That's completely no good for a postmaster SIGHUP cycle, not to mention
+ * SSL context reload in an EXEC_BACKEND postmaster child.  So override it
+ * with this dummy function that just returns an empty passphrase,
+ * guaranteeing failure.  Later we might think about collecting a passphrase
+ * at server start and feeding it to OpenSSL repeatedly, but we'd still
+ * need this callback for that.
+ */
+static int
+ssl_passwd_cb(char *buf, int size, int rwflag, void *userdata)
+{
+	ereport(LOG,
+			(errcode(ERRCODE_CONFIG_FILE_ERROR),
+			 errmsg("server's private key file requires a passphrase")));
+	Assert(size > 0);
+	buf[0] = '\0';
+	return 0;
+}
+
 /*
  *	Certificate verification callback
  *