Add an enforcement mechanism for global object names in regression tests.

In commit 18555b13 we tentatively established a rule that regression
tests should use names containing "regression" for databases, and names
starting with "regress_" for all other globally-visible object names, so
as to circumscribe the side-effects that "make installcheck" could have
on an existing installation.

This commit adds a simple enforcement mechanism for that rule: if the code
is compiled with ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS defined, it
will emit a warning (not an error) whenever a database, role, tablespace,
subscription, or replication origin name is created that doesn't obey the
rule.  Running one or more buildfarm members with that symbol defined
should be enough to catch new violations, at least in the regular
regression tests.  Most TAP tests wouldn't notice such warnings, but
that's actually fine because TAP tests don't execute against an existing
server anyway.

Since it's already the case that running src/test/modules/ tests in
installcheck mode is deprecated, we can use that as a home for tests
that seem unsafe to run against an existing server, such as tests that
might have side-effects on existing roles.  Document that (though this
commit doesn't in itself make it any less safe than before).

Update regress.sgml to define these restrictions more clearly, and
to clean up assorted lack-of-up-to-date-ness in its descriptions of
the available regression tests.

Discussion: https://postgr.es/m/16638.1468620817@sss.pgh.pa.us

doc/src/sgml/regress.sgml

@@ -47,7 +47,7 @@ make check
 <screen>
 <computeroutput>
 =======================
- All 115 tests passed.
+ All 193 tests passed.
 =======================
 </computeroutput>
 </screen>
@@ -98,7 +98,7 @@ make MAX_CONNECTIONS=10 check
 
   <para>
    To run the tests after installation (see <xref linkend="installation"/>),
-   initialize a data area and start the
+   initialize a data directory and start the
    server as explained in <xref linkend="runtime"/>, then type:
 <screen>
 make installcheck
@@ -116,10 +116,10 @@ make installcheck-parallel
 
   <para>
    The tests will also transiently create some cluster-wide objects, such as
-   roles and tablespaces.  These objects will have names beginning with
-   <literal>regress_</literal>.  Beware of using <literal>installcheck</literal>
-   mode in installations that have any actual users or tablespaces named
-   that way.
+   roles, tablespaces, and subscriptions.  These objects will have names
+   beginning with <literal>regress_</literal>.  Beware of
+   using <literal>installcheck</literal> mode with an installation that has
+   any actual global objects named that way.
   </para>
   </sect2>
 
@@ -130,7 +130,7 @@ make installcheck-parallel
    The <literal>make check</literal> and <literal>make installcheck</literal> commands
    run only the <quote>core</quote> regression tests, which test built-in
    functionality of the <productname>PostgreSQL</productname> server.  The source
-   distribution also contains additional test suites, most of them having
+   distribution contains many additional test suites, most of them having
    to do with add-on functionality such as optional procedural languages.
   </para>
 
@@ -146,9 +146,24 @@ make installcheck-world
    already-installed server, respectively, just as previously explained
    for <literal>make check</literal> and <literal>make installcheck</literal>.  Other
    considerations are the same as previously explained for each method.
-   Note that <literal>make check-world</literal> builds a separate temporary
-   installation tree for each tested module, so it requires a great deal
-   more time and disk space than <literal>make installcheck-world</literal>.
+   Note that <literal>make check-world</literal> builds a separate instance
+   (temporary data directory) for each tested module, so it requires more
+   time and disk space than <literal>make installcheck-world</literal>.
+  </para>
+
+  <para>
+   On a modern machine with multiple CPU cores and no tight operating-system
+   limits, you can make things go substantially faster with parallelism.
+   The recipe that most PostgreSQL developers actually use for running all
+   tests is something like
+<screen>
+make check-world -j8 >/dev/null
+</screen>
+   with a <option>-j</option> limit near to or a bit more than the number
+   of available cores.  Discarding <systemitem>stdout</systemitem>
+   eliminates chatter that's not interesting when you just want to verify
+   success.  (In case of failure, the <systemitem>stderr</systemitem>
+   messages are usually enough to determine where to look closer.)
   </para>
 
   <para>
@@ -166,8 +181,7 @@ make installcheck-world
   <itemizedlist>
    <listitem>
     <para>
-     Regression tests for optional procedural languages (other than
-     <application>PL/pgSQL</application>, which is tested by the core tests).
+     Regression tests for optional procedural languages.
      These are located under <filename>src/pl</filename>.
     </para>
    </listitem>
@@ -184,6 +198,13 @@ make installcheck-world
      located in <filename>src/interfaces/ecpg/test</filename>.
     </para>
    </listitem>
+   <listitem>
+    <para>
+     Tests for core-supported authentication methods,
+     located in <filename>src/test/authentication</filename>.
+     (See below for additional authentication-related tests.)
+    </para>
+   </listitem>
    <listitem>
     <para>
      Tests stressing behavior of concurrent sessions,
@@ -192,21 +213,36 @@ make installcheck-world
    </listitem>
    <listitem>
     <para>
-     Tests of client programs under <filename>src/bin</filename>.  See
-     also <xref linkend="regress-tap"/>.
+     Tests for crash recovery and physical replication,
+     located in <filename>src/test/recovery</filename>.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     Tests for logical replication,
+     located in <filename>src/test/subscription</filename>.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     Tests of client programs, located under <filename>src/bin</filename>.
     </para>
    </listitem>
   </itemizedlist>
 
   <para>
-   When using <literal>installcheck</literal> mode, these tests will destroy any
-   existing databases named <literal>pl_regression</literal>,
-   <literal>contrib_regression</literal>, <literal>isolation_regression</literal>,
-   <literal>ecpg1_regression</literal>, or <literal>ecpg2_regression</literal>, as well as
-   <literal>regression</literal>.
+   When using <literal>installcheck</literal> mode, these tests will create
+   and destroy test databases whose names
+   include <literal>regression</literal>, for
+   example <literal>pl_regression</literal>
+   or <literal>contrib_regression</literal>.  Beware of
+   using <literal>installcheck</literal> mode with an installation that has
+   any non-test databases named that way.
   </para>
 
   <para>
+   Some of these auxiliary test suites use the TAP infrastructure explained
+   in <xref linkend="regress-tap"/>.
    The TAP-based tests are run only when PostgreSQL was configured with the
    option <option>--enable-tap-tests</option>.  This is recommended for
    development, but can be omitted if there is no suitable Perl installation.
@@ -259,6 +295,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
    configuration are not run even if they are mentioned in
    <varname>PG_TEST_EXTRA</varname>.
   </para>
+
+  <para>
+   In addition, there are tests in <filename>src/test/modules</filename>
+   which will be run by <literal>make check-world</literal> but not
+   by <literal>make installcheck-world</literal>.  This is because they
+   install non-production extensions or have other side-effects that are
+   considered undesirable for a production installation.  You can
+   use <literal>make install</literal> and <literal>make
+   installcheck</literal> in one of those subdirectories if you wish,
+   but it's not recommended to do so with a non-test server.
+  </para>
   </sect2>
 
   <sect2>
@@ -737,6 +784,26 @@ make check PROVE_TESTS='t/001_test1.pl t/003_test3.pl'
     The TAP tests require the Perl module <literal>IPC::Run</literal>.
     This module is available from CPAN or an operating system package.
    </para>
+
+   <para>
+    Generically speaking, the TAP tests will test the executables in a
+    previously-installed installation tree if you say <literal>make
+    installcheck</literal>, or will build a new local installation tree from
+    current sources if you say <literal>make check</literal>.  In either
+    case they will initialize a local instance (data directory) and
+    transiently run a server in it.  Some of these tests run more than one
+    server.  Thus, these tests can be fairly resource-intensive.
+   </para>
+
+   <para>
+    It's important to realize that the TAP tests will start test server(s)
+    even when you say <literal>make installcheck</literal>; this is unlike
+    the traditional non-TAP testing infrastructure, which expects to use an
+    already-running test server in that case.  Some PostgreSQL
+    subdirectories contain both traditional-style and TAP-style tests,
+    meaning that <literal>make installcheck</literal> will produce a mix of
+    results from temporary servers and the already-running test server.
+   </para>
   </sect1>
 
   <sect1 id="regress-coverage">

src/backend/commands/alter.c

@@ -274,6 +274,12 @@ AlterObjectRename_internal(Relation rel, Oid objectId, const char *new_name)
 		if (SearchSysCacheExists2(SUBSCRIPTIONNAME, MyDatabaseId,
 								  CStringGetDatum(new_name)))
 			report_name_conflict(classId, new_name);
+
+		/* Also enforce regression testing naming rules, if enabled */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+		if (strncmp(new_name, "regress_", 8) != 0)
+			elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
+#endif
 	}
 	else if (nameCacheId >= 0)
 	{

src/backend/commands/dbcommands.c

@@ -470,6 +470,16 @@ createdb(ParseState *pstate, const CreatedbStmt *stmt)
 		/* Note there is no additional permission check in this path */
 	}
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for database names are violated.  But don't complain during
+	 * initdb.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (IsUnderPostmaster && strstr(dbname, "regression") == NULL)
+		elog(WARNING, "databases created by regression test cases should have names including \"regression\"");
+#endif
+
 	/*
 	 * Check for db name conflict.  This is just to give a more friendly error
 	 * message than "unique index violation".  There's a race condition but
@@ -1008,6 +1018,15 @@ RenameDatabase(const char *oldname, const char *newname)
 				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
 				 errmsg("permission denied to rename database")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for database names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strstr(newname, "regression") == NULL)
+		elog(WARNING, "databases created by regression test cases should have names including \"regression\"");
+#endif
+
 	/*
 	 * Make sure the new name doesn't exist.  See notes for same error in
 	 * CREATE DATABASE.

src/backend/commands/subscriptioncmds.c

@@ -357,6 +357,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
 				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
 				 (errmsg("must be superuser to create subscriptions"))));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for subscription names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(stmt->subname, "regress_", 8) != 0)
+		elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	rel = table_open(SubscriptionRelationId, RowExclusiveLock);
 
 	/* Check if name is used */

src/backend/commands/tablespace.c

@@ -307,6 +307,15 @@ CreateTableSpace(CreateTableSpaceStmt *stmt)
 						stmt->tablespacename),
 				 errdetail("The prefix \"pg_\" is reserved for system tablespaces.")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for tablespace names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(stmt->tablespacename, "regress_", 8) != 0)
+		elog(WARNING, "tablespaces created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	/*
 	 * Check that there is no other tablespace by this name.  (The unique
 	 * index would catch this anyway, but might as well give a friendlier
@@ -957,6 +966,15 @@ RenameTableSpace(const char *oldname, const char *newname)
 				 errmsg("unacceptable tablespace name \"%s\"", newname),
 				 errdetail("The prefix \"pg_\" is reserved for system tablespaces.")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for tablespace names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(newname, "regress_", 8) != 0)
+		elog(WARNING, "tablespaces created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	/* Make sure the new name doesn't exist */
 	ScanKeyInit(&entry[0],
 				Anum_pg_tablespace_spcname,

src/backend/commands/user.c

@@ -326,6 +326,15 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
 						stmt->role),
 				 errdetail("Role names starting with \"pg_\" are reserved.")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for role names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(stmt->role, "regress_", 8) != 0)
+		elog(WARNING, "roles created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	/*
 	 * Check the pg_authid relation to be certain the role doesn't already
 	 * exist.
@@ -1212,6 +1221,15 @@ RenameRole(const char *oldname, const char *newname)
 						newname),
 				 errdetail("Role names starting with \"pg_\" are reserved.")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for role names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(newname, "regress_", 8) != 0)
+		elog(WARNING, "roles created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	/* make sure the new name doesn't exist */
 	if (SearchSysCacheExists1(AUTHNAME, CStringGetDatum(newname)))
 		ereport(ERROR,

src/backend/replication/logical/origin.c

@@ -1238,6 +1238,15 @@ pg_replication_origin_create(PG_FUNCTION_ARGS)
 						name),
 				 errdetail("Origin names starting with \"pg_\" are reserved.")));
 
+	/*
+	 * If built with appropriate switch, whine when regression-testing
+	 * conventions for replication origin names are violated.
+	 */
+#ifdef ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS
+	if (strncmp(name, "regress_", 8) != 0)
+		elog(WARNING, "replication origins created by regression test cases should have names starting with \"regress_\"");
+#endif
+
 	roident = replorigin_create(name);
 
 	pfree(name);

src/test/modules/README

@@ -6,6 +6,13 @@ intended for testing PostgreSQL and/or to serve as example code. The extensions
 here aren't intended to be installed in a production server and aren't suitable
 for "real work".
 
+Furthermore, while you can do "make install" and "make installcheck" in
+this directory or its children, it is NOT ADVISABLE to do so with a server
+containing valuable data.  Some of these tests may have undesirable
+side-effects on roles or other global objects within the tested server.
+"make installcheck-world" at the top level does not recurse into this
+directory.
+
 Most extensions have their own pg_regress tests or isolationtester specs. Some
 are also used by tests elsewhere in the tree.