Generate backup manifests for base backups, and validate them.

A manifest is a JSON document which includes (1) the file name, size,
last modification time, and an optional checksum for each file backed
up, (2) timelines and LSNs for whatever WAL will need to be replayed
to make the backup consistent, and (3) a checksum for the manifest
itself. By default, we use CRC-32C when checksumming data files,
because we are trying to detect corruption and user error, not foil an
adversary. However, pg_basebackup and the server-side BASE_BACKUP
command now have options to select a different algorithm, so users
wanting a cryptographic hash function can select SHA-224, SHA-256,
SHA-384, or SHA-512. Users not wanting file checksums at all can
disable them, or disable generating of the backup manifest altogether.
Using a cryptographic hash function in place of CRC-32C consumes
significantly more CPU cycles, which may slow down backups in some
cases.

A new tool called pg_validatebackup can validate a backup against the
manifest. If no checksums are present, it can still check that the
right files exist and that they have the expected sizes. If checksums
are present, it can also verify that each file has the expected
checksum. Additionally, it calls pg_waldump to verify that the
expected WAL files are present and parseable. Only plain format
backups can be validated directly, but tar format backups can be
validated after extracting them.

Robert Haas, with help, ideas, review, and testing from David Steele,
Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, Tushar
Ahuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, Jeevan
Chalke, Amit Kapila, Andres Freund, and Noah Misch.

Discussion: http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com

doc/src/sgml/protocol.sgml

@@ -2466,7 +2466,7 @@ The commands accepted in replication mode are:
   </varlistentry>
 
   <varlistentry id="protocol-replication-base-backup" xreflabel="BASE_BACKUP">
-    <term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ]
+    <term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ] [ <literal>MANIFEST</literal> <replaceable>manifest_option</replaceable> ] [ <literal>MANIFEST_CHECKSUMS</literal> <replaceable>checksum_algorithm</replaceable> ]
      <indexterm><primary>BASE_BACKUP</primary></indexterm>
     </term>
     <listitem>
@@ -2576,6 +2576,41 @@ The commands accepted in replication mode are:
          </para>
         </listitem>
        </varlistentry>
+
+       <varlistentry>
+        <term><literal>MANIFEST</literal></term>
+        <listitem>
+         <para>
+          When this option is specified with a value of <literal>yes</literal>
+          or <literal>force-escape</literal>, a backup manifest is created
+          and sent along with the backup.  The manifest is a list of every
+          file present in the backup with the exception of any WAL files that
+          may be included. It also stores the size, last modification time, and
+          an optional checksum for each file.
+          A value of <literal>force-escape</literal> forces all filenames
+          to be hex-encoded; otherwise, this type of encoding is performed only
+          for files whose names are non-UTF8 octet sequences.
+          <literal>force-escape</literal> is intended primarily for testing
+          purposes, to be sure that clients which read the backup manifest
+          can handle this case. For compatibility with previous releases,
+          the default is <literal>MANIFEST 'no'</literal>.
+         </para>
+        </listitem>
+       </varlistentry>
+
+       <varlistentry>
+        <term><literal>MANIFEST_CHECKSUMS</literal></term>
+        <listitem>
+         <para>
+          Specifies the algorithm that should be applied to each file included
+          in the backup manifest. Currently, the available
+          algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
+          <literal>SHA224</literal>, <literal>SHA256</literal>,
+          <literal>SHA384</literal>, and <literal>SHA512</literal>.
+          The default is <literal>CRC32C</literal>.
+         </para>
+        </listitem>
+       </varlistentry>
       </variablelist>
      </para>
      <para>

doc/src/sgml/ref/allfiles.sgml

@@ -211,6 +211,7 @@ Complete list of usable sgml source files in this directory.
 <!ENTITY pgResetwal         SYSTEM "pg_resetwal.sgml">
 <!ENTITY pgRestore          SYSTEM "pg_restore.sgml">
 <!ENTITY pgRewind           SYSTEM "pg_rewind.sgml">
+<!ENTITY pgValidateBackup   SYSTEM "pg_validatebackup.sgml">
 <!ENTITY pgtestfsync        SYSTEM "pgtestfsync.sgml">
 <!ENTITY pgtesttiming       SYSTEM "pgtesttiming.sgml">
 <!ENTITY pgupgrade          SYSTEM "pgupgrade.sgml">

doc/src/sgml/ref/pg_basebackup.sgml

@@ -561,6 +561,70 @@ PostgreSQL documentation
        </para>
       </listitem>
      </varlistentry>
+
+     <varlistentry>
+      <term><option>--no-manifest</option></term>
+      <listitem>
+       <para>
+        Disables generation of a backup manifest. If this option is not
+        specified, the server will generate and send a backup manifest
+        which can be verified using <xref linkend="app-pgvalidatebackup" />.
+        The manifest is a list of every file present in the backup with the
+        exception of any WAL files that may be included. It also stores the
+        size, last modification time, and an optional checksum for each file.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><option>--manifest-force-encode</option></term>
+      <listitem>
+       <para>
+        Forces all filenames in the backup manifest to be hex-encoded.
+        If this option is not specified, only non-UTF8 filenames are
+        hex-encoded. This option is mostly intended to test that tools which
+        read a backup manifest file properly handle this case.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><option>--manifest-checksums=<replaceable class="parameter">algorithm</replaceable></option></term>
+      <listitem>
+       <para>
+        Specifies the checksum algorithm that should be applied to each file
+        included in the backup manifest. Currently, the available
+        algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
+        <literal>SHA224</literal>, <literal>SHA256</literal>,
+        <literal>SHA384</literal>, and <literal>SHA512</literal>.
+        The default is <literal>CRC32C</literal>.
+       </para>
+       <para>
+        If <literal>NONE</literal> is selected, the backup manifest will
+        not contain any checksums. Otherwise, it will contain a checksum
+        of each file in the backup using the specified algorithm. In addition,
+        the manifest will always contain a <literal>SHA256</literal>
+        checksum of its own contents. The <literal>SHA</literal> algorithms
+        are significantly more CPU-intensive than <literal>CRC32C</literal>,
+        so selecting one of them may increase the time required to complete
+        the backup.
+       </para>
+       <para>
+        Using a SHA hash function provides a cryptographically secure digest
+        of each file for users who wish to verify that the backup has not been
+        tampered with, while the CRC32C algorithm provides a checksum which is
+        much faster to calculate and good at catching errors due to accidental
+        changes but is not resistant to targeted modifications.  Note that, to
+        be useful against an adversary who has access to the backup, the backup
+        manifest would need to be stored securely elsewhere or otherwise
+        verified not to have been modified since the backup was taken.
+       </para>
+       <para>
+        <xref linkend="app-pgvalidatebackup" /> can be used to check the
+        integrity of a backup against the backup manifest.
+       </para>
+      </listitem>
+     </varlistentry>
     </variablelist>
    </para>
 

doc/src/sgml/reference.sgml

@@ -255,6 +255,7 @@
    &pgReceivewal;
    &pgRecvlogical;
    &pgRestore;
+   &pgValidateBackup;
    &psqlRef;
    &reindexdb;
    &vacuumdb;

src/backend/access/transam/xlog.c

@@ -10642,7 +10642,8 @@ do_pg_start_backup(const char *backupidstr, bool fast, TimeLineID *starttli_p,
 			ti->oid = pstrdup(de->d_name);
 			ti->path = pstrdup(buflinkpath.data);
 			ti->rpath = relpath ? pstrdup(relpath) : NULL;
-			ti->size = infotbssize ? sendTablespace(fullpath, true) : -1;
+			ti->size = infotbssize ?
+				sendTablespace(fullpath, ti->oid, true, NULL) : -1;
 
 			if (tablespaces)
 				*tablespaces = lappend(*tablespaces, ti);

src/backend/replication/repl_gram.y

@@ -87,6 +87,8 @@ static SQLCmd *make_sqlcmd(void);
 %token K_EXPORT_SNAPSHOT
 %token K_NOEXPORT_SNAPSHOT
 %token K_USE_SNAPSHOT
+%token K_MANIFEST
+%token K_MANIFEST_CHECKSUMS
 
 %type <node>	command
 %type <node>	base_backup start_replication start_logical_replication
@@ -156,6 +158,7 @@ var_name:	IDENT	{ $$ = $1; }
 /*
  * BASE_BACKUP [LABEL '<label>'] [PROGRESS] [FAST] [WAL] [NOWAIT]
  * [MAX_RATE %d] [TABLESPACE_MAP] [NOVERIFY_CHECKSUMS]
+ * [MANIFEST %s] [MANIFEST_CHECKSUMS %s]
  */
 base_backup:
 			K_BASE_BACKUP base_backup_opt_list
@@ -214,6 +217,16 @@ base_backup_opt:
 				  $$ = makeDefElem("noverify_checksums",
 								   (Node *)makeInteger(true), -1);
 				}
+			| K_MANIFEST SCONST
+				{
+				  $$ = makeDefElem("manifest",
+								   (Node *)makeString($2), -1);
+				}
+			| K_MANIFEST_CHECKSUMS SCONST
+				{
+				  $$ = makeDefElem("manifest_checksums",
+								   (Node *)makeString($2), -1);
+				}
 			;
 
 create_replication_slot:

src/backend/replication/repl_scanner.l

@@ -107,6 +107,8 @@ EXPORT_SNAPSHOT		{ return K_EXPORT_SNAPSHOT; }
 NOEXPORT_SNAPSHOT	{ return K_NOEXPORT_SNAPSHOT; }
 USE_SNAPSHOT		{ return K_USE_SNAPSHOT; }
 WAIT				{ return K_WAIT; }
+MANIFEST			{ return K_MANIFEST; }
+MANIFEST_CHECKSUMS	{ return K_MANIFEST_CHECKSUMS; }
 
 ","				{ return ','; }
 ";"				{ return ';'; }

src/backend/replication/walsender.c

@@ -315,6 +315,8 @@ WalSndErrorCleanup(void)
 
 	replication_active = false;
 
+	WalSndResourceCleanup(false);
+
 	if (got_STOPPING || got_SIGUSR2)
 		proc_exit(0);
 
@@ -322,6 +324,34 @@ WalSndErrorCleanup(void)
 	WalSndSetState(WALSNDSTATE_STARTUP);
 }
 
+/*
+ * Clean up any ResourceOwner we created.
+ */
+void
+WalSndResourceCleanup(bool isCommit)
+{
+	ResourceOwner	resowner;
+
+	if (CurrentResourceOwner == NULL)
+		return;
+
+	/*
+	 * Deleting CurrentResourceOwner is not allowed, so we must save a
+	 * pointer in a local variable and clear it first.
+	 */
+	resowner = CurrentResourceOwner;
+	CurrentResourceOwner = NULL;
+
+	/* Now we can release resources and delete it. */
+	ResourceOwnerRelease(resowner,
+						 RESOURCE_RELEASE_BEFORE_LOCKS, isCommit, true);
+	ResourceOwnerRelease(resowner,
+						 RESOURCE_RELEASE_LOCKS, isCommit, true);
+	ResourceOwnerRelease(resowner,
+						 RESOURCE_RELEASE_AFTER_LOCKS, isCommit, true);
+	ResourceOwnerDelete(resowner);
+}
+
 /*
  * Handle a client's connection abort in an orderly manner.
  */

src/bin/Makefile

@@ -27,6 +27,7 @@ SUBDIRS = \
 	pg_test_fsync \
 	pg_test_timing \
 	pg_upgrade \
+	pg_validatebackup \
 	pg_waldump \
 	pgbench \
 	psql \

src/bin/pg_basebackup/pg_basebackup.c

@@ -88,6 +88,12 @@ typedef struct UnpackTarState
 	FILE	   *file;
 } UnpackTarState;
 
+typedef struct WriteManifestState
+{
+	char		filename[MAXPGPATH];
+	FILE	   *file;
+} WriteManifestState;
+
 typedef void (*WriteDataCallback) (size_t nbytes, char *buf,
 								   void *callback_data);
 
@@ -136,6 +142,9 @@ static bool temp_replication_slot = true;
 static bool create_slot = false;
 static bool no_slot = false;
 static bool verify_checksums = true;
+static bool manifest = true;
+static bool manifest_force_encode = false;
+static char *manifest_checksums = NULL;
 
 static bool success = false;
 static bool made_new_pgdata = false;
@@ -181,6 +190,12 @@ static void ReceiveTarCopyChunk(size_t r, char *copybuf, void *callback_data);
 static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
 static void ReceiveTarAndUnpackCopyChunk(size_t r, char *copybuf,
 										 void *callback_data);
+static void ReceiveBackupManifest(PGconn *conn);
+static void ReceiveBackupManifestChunk(size_t r, char *copybuf,
+									   void *callback_data);
+static void ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf);
+static void ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
+											   void *callback_data);
 static void BaseBackup(void);
 
 static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -388,6 +403,11 @@ usage(void)
 	printf(_("      --no-verify-checksums\n"
 			 "                         do not verify checksums\n"));
 	printf(_("      --no-estimate-size do not estimate backup size in server side\n"));
+	printf(_("      --no-manifest      suppress generation of backup manifest\n"));
+	printf(_("      --manifest-force-encode\n"
+			 "                         hex encode all filenames in manifest\n"));
+	printf(_("      --manifest-checksums=SHA{224,256,384,512}|CRC32C|NONE\n"
+			 "                         use algorithm for manifest checksums\n"));
 	printf(_("  -?, --help             show this help, then exit\n"));
 	printf(_("\nConnection options:\n"));
 	printf(_("  -d, --dbname=CONNSTR   connection string\n"));
@@ -1186,6 +1206,31 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
 		}
 	}
 
+	/*
+	 * Normally, we emit the backup manifest as a separate file, but when
+	 * we're writing a tarfile to stdout, we don't have that option, so
+	 * include it in the one tarfile we've got.
+	 */
+	if (strcmp(basedir, "-") == 0)
+	{
+		char		header[512];
+		PQExpBufferData	buf;
+
+		initPQExpBuffer(&buf);
+		ReceiveBackupManifestInMemory(conn, &buf);
+		if (PQExpBufferDataBroken(buf))
+		{
+			pg_log_error("out of memory");
+			exit(1);
+		}
+		tarCreateHeader(header, "backup_manifest", NULL, buf.len,
+						pg_file_create_mode, 04000, 02000,
+						time(NULL));
+		writeTarData(&state, header, sizeof(header));
+		writeTarData(&state, buf.data, buf.len);
+		termPQExpBuffer(&buf);
+	}
+
 	/* 2 * 512 bytes empty data at end of file */
 	writeTarData(&state, zerobuf, sizeof(zerobuf));
 
@@ -1657,6 +1702,64 @@ ReceiveTarAndUnpackCopyChunk(size_t r, char *copybuf, void *callback_data)
 	}							/* continuing data in existing file */
 }
 
+/*
+ * Receive the backup manifest file and write it out to a file.
+ */
+static void
+ReceiveBackupManifest(PGconn *conn)
+{
+	WriteManifestState state;
+
+	snprintf(state.filename, sizeof(state.filename),
+			 "%s/backup_manifest.tmp", basedir);
+	state.file = fopen(state.filename, "wb");
+	if (state.file == NULL)
+	{
+		pg_log_error("could not create file \"%s\": %m", state.filename);
+		exit(1);
+	}
+
+	ReceiveCopyData(conn, ReceiveBackupManifestChunk, &state);
+
+	fclose(state.file);
+}
+
+/*
+ * Receive one chunk of the backup manifest file and write it out to a file.
+ */
+static void
+ReceiveBackupManifestChunk(size_t r, char *copybuf, void *callback_data)
+{
+	WriteManifestState *state = callback_data;
+
+	if (fwrite(copybuf, r, 1, state->file) != 1)
+	{
+		pg_log_error("could not write to file \"%s\": %m", state->filename);
+		exit(1);
+	}
+}
+
+/*
+ * Receive the backup manifest file and write it out to a file.
+ */
+static void
+ReceiveBackupManifestInMemory(PGconn *conn, PQExpBuffer buf)
+{
+	ReceiveCopyData(conn, ReceiveBackupManifestInMemoryChunk, buf);
+}
+
+/*
+ * Receive one chunk of the backup manifest file and write it out to a file.
+ */
+static void
+ReceiveBackupManifestInMemoryChunk(size_t r, char *copybuf,
+								   void *callback_data)
+{
+	PQExpBuffer buf = callback_data;
+
+	appendPQExpBuffer(buf, copybuf, r);
+}
+
 static void
 BaseBackup(void)
 {
@@ -1667,6 +1770,8 @@ BaseBackup(void)
 	char	   *basebkp;
 	char		escaped_label[MAXPGPATH];
 	char	   *maxrate_clause = NULL;
+	char	   *manifest_clause;
+	char	   *manifest_checksums_clause = "";
 	int			i;
 	char		xlogstart[64];
 	char		xlogend[64];
@@ -1674,6 +1779,7 @@ BaseBackup(void)
 				maxServerMajor;
 	int			serverVersion,
 				serverMajor;
+	int			writing_to_stdout;
 
 	Assert(conn != NULL);
 
@@ -1728,6 +1834,33 @@ BaseBackup(void)
 	if (maxrate > 0)
 		maxrate_clause = psprintf("MAX_RATE %u", maxrate);
 
+	if (manifest)
+	{
+		if (serverMajor < 1300)
+		{
+			const char *serverver = PQparameterStatus(conn, "server_version");
+
+			pg_log_error("backup manifests are not supported by server version %s",
+						 serverver ? serverver : "'unknown'");
+			exit(1);
+		}
+
+		if (manifest_force_encode)
+			manifest_clause = "MANIFEST 'force-encode'";
+		else
+			manifest_clause = "MANIFEST 'yes'";
+		if (manifest_checksums != NULL)
+			manifest_checksums_clause = psprintf("MANIFEST_CHECKSUMS '%s'",
+												 manifest_checksums);
+	}
+	else
+	{
+		if (serverMajor < 1300)
+			manifest_clause = "";
+		else
+			manifest_clause = "MANIFEST 'no'";
+	}
+
 	if (verbose)
 		pg_log_info("initiating base backup, waiting for checkpoint to complete");
 
@@ -1741,7 +1874,7 @@ BaseBackup(void)
 	}
 
 	basebkp =
-		psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s %s",
+		psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s %s %s %s",
 				 escaped_label,
 				 estimatesize ? "PROGRESS" : "",
 				 includewal == FETCH_WAL ? "WAL" : "",
@@ -1749,7 +1882,9 @@ BaseBackup(void)
 				 includewal == NO_WAL ? "" : "NOWAIT",
 				 maxrate_clause ? maxrate_clause : "",
 				 format == 't' ? "TABLESPACE_MAP" : "",
-				 verify_checksums ? "" : "NOVERIFY_CHECKSUMS");
+				 verify_checksums ? "" : "NOVERIFY_CHECKSUMS",
+				 manifest_clause,
+				 manifest_checksums_clause);
 
 	if (PQsendQuery(conn, basebkp) == 0)
 	{
@@ -1837,7 +1972,8 @@ BaseBackup(void)
 	/*
 	 * When writing to stdout, require a single tablespace
 	 */
-	if (format == 't' && strcmp(basedir, "-") == 0 && PQntuples(res) > 1)
+	writing_to_stdout = format == 't' && strcmp(basedir, "-") == 0;
+	if (writing_to_stdout && PQntuples(res) > 1)
 	{
 		pg_log_error("can only write single tablespace to stdout, database has %d",
 					 PQntuples(res));
@@ -1866,6 +2002,19 @@ BaseBackup(void)
 			ReceiveAndUnpackTarFile(conn, res, i);
 	}							/* Loop over all tablespaces */
 
+	/*
+	 * Now receive backup manifest, if appropriate.
+	 *
+	 * If we're writing a tarfile to stdout, ReceiveTarFile will have already
+	 * processed the backup manifest and included it in the output tarfile.
+	 * Such a configuration doesn't allow for writing multiple files.
+	 *
+	 * If we're talking to an older server, it won't send a backup manifest,
+	 * so don't try to receive one.
+	 */
+	if (!writing_to_stdout && manifest)
+		ReceiveBackupManifest(conn);
+
 	if (showprogress)
 	{
 		progress_report(PQntuples(res), NULL, true);
@@ -2031,6 +2180,29 @@ BaseBackup(void)
 		}
 	}
 
+	/*
+	 * After synchronizing data to disk, perform a durable rename of
+	 * backup_manifest.tmp to backup_manifest, if we wrote such a file. This
+	 * way, a failure or system crash before we reach this point will leave us
+	 * without a backup_manifest file, decreasing the chances that a directory
+	 * we leave behind will be mistaken for a valid backup.
+	 */
+	if (!writing_to_stdout && manifest)
+	{
+		char		tmp_filename[MAXPGPATH];
+		char		filename[MAXPGPATH];
+
+		if (verbose)
+			pg_log_info("renaming backup_manifest.tmp to backup_manifest");
+
+		snprintf(tmp_filename, MAXPGPATH, "%s/backup_manifest.tmp", basedir);
+		snprintf(filename, MAXPGPATH, "%s/backup_manifest", basedir);
+
+		/* durable_rename emits its own log message in case of failure */
+		if (durable_rename(tmp_filename, filename) != 0)
+			exit(1);
+	}
+
 	if (verbose)
 		pg_log_info("base backup completed");
 }
@@ -2069,6 +2241,9 @@ main(int argc, char **argv)
 		{"no-slot", no_argument, NULL, 2},
 		{"no-verify-checksums", no_argument, NULL, 3},
 		{"no-estimate-size", no_argument, NULL, 4},
+		{"no-manifest", no_argument, NULL, 5},
+		{"manifest-force-encode", no_argument, NULL, 6},
+		{"manifest-checksums", required_argument, NULL, 7},
 		{NULL, 0, NULL, 0}
 	};
 	int			c;
@@ -2096,7 +2271,7 @@ main(int argc, char **argv)
 
 	atexit(cleanup_directories_atexit);
 
-	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvP",
+	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWkvPm:",
 							long_options, &option_index)) != -1)
 	{
 		switch (c)
@@ -2240,6 +2415,15 @@ main(int argc, char **argv)
 			case 4:
 				estimatesize = false;
 				break;
+			case 5:
+				manifest = false;
+				break;
+			case 6:
+				manifest_force_encode = true;
+				break;
+			case 7:
+				manifest_checksums = pg_strdup(optarg);
+				break;
 			default:
 
 				/*
@@ -2370,6 +2554,22 @@ main(int argc, char **argv)
 		exit(1);
 	}
 
+	if (!manifest && manifest_checksums != NULL)
+	{
+		pg_log_error("--no-manifest and --manifest-checksums are incompatible options");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
+	if (!manifest && manifest_force_encode)
+	{
+		pg_log_error("--no-manifest and --manifest-force-encode are incompatible options");
+		fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
+				progname);
+		exit(1);
+	}
+
 	/* connection in replication mode to server */
 	conn = GetConnection();
 	if (!conn)

src/bin/pg_basebackup/t/010_pg_basebackup.pl

@@ -6,7 +6,7 @@ use File::Basename qw(basename dirname);
 use File::Path qw(rmtree);
 use PostgresNode;
 use TestLib;
-use Test::More tests => 107;
+use Test::More tests => 109;
 
 program_help_ok('pg_basebackup');
 program_version_ok('pg_basebackup');
@@ -104,6 +104,7 @@ foreach my $filename (@tempRelationFiles)
 $node->command_ok([ 'pg_basebackup', '-D', "$tempdir/backup", '-X', 'none' ],
 	'pg_basebackup runs');
 ok(-f "$tempdir/backup/PG_VERSION", 'backup was created');
+ok(-f "$tempdir/backup/backup_manifest", 'backup manifest included');
 
 # Permissions on backup should be default
 SKIP:
@@ -160,11 +161,12 @@ rmtree("$tempdir/backup");
 
 $node->command_ok(
 	[
-		'pg_basebackup', '-D', "$tempdir/backup2", '--waldir',
-		"$tempdir/xlog2"
+		'pg_basebackup', '-D', "$tempdir/backup2", '--no-manifest',
+		'--waldir', "$tempdir/xlog2"
 	],
 	'separate xlog directory');
 ok(-f "$tempdir/backup2/PG_VERSION", 'backup was created');
+ok(! -f "$tempdir/backup2/backup_manifest", 'manifest was suppressed');
 ok(-d "$tempdir/xlog2/",             'xlog directory was created');
 rmtree("$tempdir/backup2");
 rmtree("$tempdir/xlog2");

src/bin/pg_validatebackup/.gitignore

+/pg_validatebackup
+/tmp_check/

src/bin/pg_validatebackup/Makefile

+# src/bin/pg_validatebackup/Makefile
+
+PGFILEDESC = "pg_validatebackup - validate a backup against a backup manifest"
+PGAPPICON = win32
+
+subdir = src/bin/pg_validatebackup
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+# We need libpq only because fe_utils does.
+LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
+
+OBJS = \
+	$(WIN32RES) \
+	parse_manifest.o \
+	pg_validatebackup.o
+
+all: pg_validatebackup
+
+pg_validatebackup: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+	$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+
+install: all installdirs
+	$(INSTALL_PROGRAM) pg_validatebackup$(X) '$(DESTDIR)$(bindir)/pg_validatebackup$(X)'
+
+installdirs:
+	$(MKDIR_P) '$(DESTDIR)$(bindir)'
+
+uninstall:
+	rm -f '$(DESTDIR)$(bindir)/pg_validatebackup$(X)'
+
+clean distclean maintainer-clean:
+	rm -f pg_validatebackup$(X) $(OBJS)
+
+check:
+	$(prove_check)
+
+installcheck:
+	$(prove_installcheck)

src/bin/pg_validatebackup/parse_manifest.h

+/*-------------------------------------------------------------------------
+ *
+ * parse_manifest.h
+ *	  Parse a backup manifest in JSON format.
+ *
+ * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_validatebackup/parse_manifest.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef PARSE_MANIFEST_H
+#define PARSE_MANIFEST_H
+
+#include "access/xlogdefs.h"
+#include "common/checksum_helper.h"
+#include "mb/pg_wchar.h"
+
+struct JsonManifestParseContext;
+typedef struct JsonManifestParseContext JsonManifestParseContext;
+
+typedef void (*json_manifest_perfile_callback)(JsonManifestParseContext *,
+								 char *pathname,
+								 size_t size, pg_checksum_type checksum_type,
+								 int checksum_length, uint8 *checksum_payload);
+typedef void (*json_manifest_perwalrange_callback)(JsonManifestParseContext *,
+								 TimeLineID tli,
+								 XLogRecPtr start_lsn, XLogRecPtr end_lsn);
+typedef void (*json_manifest_error_callback)(JsonManifestParseContext *,
+								 char *fmt, ...) pg_attribute_printf(2, 3);
+
+struct JsonManifestParseContext
+{
+	void	   *private_data;
+	json_manifest_perfile_callback perfile_cb;
+	json_manifest_perwalrange_callback perwalrange_cb;
+	json_manifest_error_callback error_cb;
+};
+
+extern void json_parse_manifest(JsonManifestParseContext *context,
+								char *buffer, size_t size);
+
+#endif

src/bin/pg_validatebackup/t/001_basic.pl

+use strict;
+use warnings;
+use TestLib;
+use Test::More tests => 16;
+
+my $tempdir = TestLib::tempdir;
+
+program_help_ok('pg_validatebackup');
+program_version_ok('pg_validatebackup');
+program_options_handling_ok('pg_validatebackup');
+
+command_fails_like(['pg_validatebackup'],
+				   qr/no backup directory specified/,
+				   'target directory must be specified');
+command_fails_like(['pg_validatebackup', $tempdir],
+				   qr/could not open file.*\/backup_manifest\"/,
+				   'pg_validatebackup requires a manifest');
+command_fails_like(['pg_validatebackup', $tempdir, $tempdir],
+				   qr/too many command-line arguments/,
+				   'multiple target directories not allowed');
+
+# create fake manifest file
+open(my $fh, '>', "$tempdir/backup_manifest") || die "open: $!";
+close($fh);
+
+# but then try to use an alternate, nonexisting manifest
+command_fails_like(['pg_validatebackup', '-m', "$tempdir/not_the_manifest",
+						$tempdir],
+				   qr/could not open file.*\/not_the_manifest\"/,
+				   'pg_validatebackup respects -m flag');

src/bin/pg_validatebackup/t/002_algorithm.pl

+# Verify that we can take and validate backups with various checksum types.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 19;
+
+my $master = get_new_node('master');
+$master->init(allows_streaming => 1);
+$master->start;
+
+for my $algorithm (qw(bogus none crc32c sha224 sha256 sha384 sha512))
+{
+	my $backup_path = $master->backup_dir . '/' . $algorithm;
+	my @backup = ('pg_basebackup', '-D', $backup_path,
+				  '--manifest-checksums', $algorithm,
+				  '--no-sync');
+	my @validate = ('pg_validatebackup', '-e', $backup_path);
+
+	# A backup with a bogus algorithm should fail.
+	if ($algorithm eq 'bogus')
+	{
+		$master->command_fails(\@backup,
+							   "backup fails with algorithm \"$algorithm\"");
+		next;
+	}
+
+	# A backup with a valid algorithm should work.
+	$master->command_ok(\@backup, "backup ok with algorithm \"$algorithm\"");
+
+	# We expect each real checksum algorithm to be mentioned on every line of
+	# the backup manifest file except the first and last; for simplicity, we
+	# just check that it shows up lots of times. When the checksum algorithm
+	# is none, we just check that the manifest exists.
+	if ($algorithm eq 'none')
+	{
+		ok(-f "$backup_path/backup_manifest", "backup manifest exists");
+	}
+	else
+	{
+		my $manifest = slurp_file("$backup_path/backup_manifest");
+		my $count_of_algorithm_in_manifest =
+			(() = $manifest =~ /$algorithm/mig);
+		cmp_ok($count_of_algorithm_in_manifest, '>', 100,
+			   "$algorithm is mentioned many times in the manifest");
+	}
+
+	# Make sure that it validates OK.
+	$master->command_ok(\@validate,
+						"validate backup with algorithm \"$algorithm\"");
+
+	# Remove backup immediately to save disk space.
+	rmtree($backup_path);
+}

src/bin/pg_validatebackup/t/003_corruption.pl

+# Verify that various forms of corruption are detected by pg_validatebackup.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 44;
+
+my $master = get_new_node('master');
+$master->init(allows_streaming => 1);
+$master->start;
+
+# Include a user-defined tablespace in the hopes of detecting problems in that
+# area.
+my $source_ts_path = TestLib::tempdir;
+$master->safe_psql('postgres', <<EOM);
+CREATE TABLE x1 (a int);
+INSERT INTO x1 VALUES (111);
+CREATE TABLESPACE ts1 LOCATION '$source_ts_path';
+CREATE TABLE x2 (a int) TABLESPACE ts1;
+INSERT INTO x1 VALUES (222);
+EOM
+
+my @scenario = (
+	{
+		'name' => 'extra_file',
+		'mutilate' => \&mutilate_extra_file,
+		'fails_like' =>
+			qr/extra_file.*present on disk but not in the manifest/
+	},
+	{
+		'name' => 'extra_tablespace_file',
+		'mutilate' => \&mutilate_extra_tablespace_file,
+		'fails_like' =>
+			qr/extra_ts_file.*present on disk but not in the manifest/
+	},
+	{
+		'name' => 'missing_file',
+		'mutilate' => \&mutilate_missing_file,
+		'fails_like' =>
+			qr/pg_xact\/0000.*present in the manifest but not on disk/
+	},
+	{
+		'name' => 'missing_tablespace',
+		'mutilate' => \&mutilate_missing_tablespace,
+		'fails_like' =>
+			qr/pg_tblspc.*present in the manifest but not on disk/
+	},
+	{
+		'name' => 'append_to_file',
+		'mutilate' => \&mutilate_append_to_file,
+		'fails_like' =>
+			qr/has size \d+ on disk but size \d+ in the manifest/
+	},
+	{
+		'name' => 'truncate_file',
+		'mutilate' => \&mutilate_truncate_file,
+		'fails_like' =>
+			qr/has size 0 on disk but size \d+ in the manifest/
+	},
+	{
+		'name' => 'replace_file',
+		'mutilate' => \&mutilate_replace_file,
+		'fails_like' => qr/checksum mismatch for file/
+	},
+	{
+		'name' => 'bad_manifest',
+		'mutilate' => \&mutilate_bad_manifest,
+		'fails_like' => qr/manifest checksum mismatch/
+	},
+	{
+		'name' => 'open_file_fails',
+		'mutilate' => \&mutilate_open_file_fails,
+		'fails_like' => qr/could not open file/,
+		'skip_on_windows' => 1
+	},
+	{
+		'name' => 'open_directory_fails',
+		'mutilate' => \&mutilate_open_directory_fails,
+		'fails_like' => qr/could not open directory/,
+		'skip_on_windows' => 1
+	},
+	{
+		'name' => 'search_directory_fails',
+		'mutilate' => \&mutilate_search_directory_fails,
+		'cleanup' => \&cleanup_search_directory_fails,
+		'fails_like' => qr/could not stat file or directory/,
+		'skip_on_windows' => 1
+	}
+);
+
+for my $scenario (@scenario)
+{
+	my $name = $scenario->{'name'};
+
+	SKIP:
+	{
+		skip "unix-style permissions not supported on Windows", 4
+			if $scenario->{'skip_on_windows'} && $windows_os;
+
+		# Take a backup and check that it validates OK.
+		my $backup_path = $master->backup_dir . '/' . $name;
+		my $backup_ts_path = TestLib::tempdir;
+		$master->command_ok(['pg_basebackup', '-D', $backup_path, '--no-sync',
+							'-T', "${source_ts_path}=${backup_ts_path}"],
+							"base backup ok");
+		command_ok(['pg_validatebackup', $backup_path ],
+				   "intact backup validated");
+
+		# Mutilate the backup in some way.
+		$scenario->{'mutilate'}->($backup_path);
+
+		# Now check that the backup no longer validates.
+		command_fails_like(['pg_validatebackup', $backup_path ],
+						   $scenario->{'fails_like'},
+						   "corrupt backup fails validation: $name");
+
+		# Run cleanup hook, if provided.
+		$scenario->{'cleanup'}->($backup_path)
+			if exists $scenario->{'cleanup'};
+
+		# Finally, use rmtree to reclaim space.
+		rmtree($backup_path);
+	}
+}
+
+sub create_extra_file
+{
+	my ($backup_path, $relative_path) = @_;
+	my $pathname = "$backup_path/$relative_path";
+	open(my $fh, '>', $pathname) || die "open $pathname: $!";
+	print $fh "This is an extra file.\n";
+	close($fh);
+}
+
+# Add a file into the root directory of the backup.
+sub mutilate_extra_file
+{
+	my ($backup_path) = @_;
+	create_extra_file($backup_path, "extra_file");
+}
+
+# Add a file inside the user-defined tablespace.
+sub mutilate_extra_tablespace_file
+{
+	my ($backup_path) = @_;
+	my ($tsoid) = grep { $_ ne '.' && $_ ne '..' }
+		 slurp_dir("$backup_path/pg_tblspc");
+	my ($catvdir) = grep { $_ ne '.' && $_ ne '..' }
+		 slurp_dir("$backup_path/pg_tblspc/$tsoid");
+	my ($tsdboid) = grep { $_ ne '.' && $_ ne '..' }
+		 slurp_dir("$backup_path/pg_tblspc/$tsoid/$catvdir");
+	create_extra_file($backup_path,
+					  "pg_tblspc/$tsoid/$catvdir/$tsdboid/extra_ts_file");
+}
+
+# Remove a file.
+sub mutilate_missing_file
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/pg_xact/0000";
+	unlink($pathname) || die "$pathname: $!";
+}
+
+# Remove the symlink to the user-defined tablespace.
+sub mutilate_missing_tablespace
+{
+	my ($backup_path) = @_;
+	my ($tsoid) = grep { $_ ne '.' && $_ ne '..' }
+		 slurp_dir("$backup_path/pg_tblspc");
+	my $pathname = "$backup_path/pg_tblspc/$tsoid";
+	if ($windows_os)
+	{
+		rmdir($pathname) || die "$pathname: $!";
+	}
+	else
+	{
+		unlink($pathname) || die "$pathname: $!";
+	}
+}
+
+# Append an additional bytes to a file.
+sub mutilate_append_to_file
+{
+	my ($backup_path) = @_;
+	append_to_file "$backup_path/global/pg_control", 'x';
+}
+
+# Truncate a file to zero length.
+sub mutilate_truncate_file
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/global/pg_control";
+	open(my $fh, '>', $pathname) || die "open $pathname: $!";
+	close($fh);
+}
+
+# Replace a file's contents without changing the length of the file. This is
+# not a particularly efficient way to do this, so we pick a file that's
+# expected to be short.
+sub mutilate_replace_file
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/PG_VERSION";
+	my $contents = slurp_file($pathname);
+	open(my $fh, '>', $pathname) || die "open $pathname: $!";
+	print $fh 'q' x length($contents);
+	close($fh);
+}
+
+# Corrupt the backup manifest.
+sub mutilate_bad_manifest
+{
+	my ($backup_path) = @_;
+	append_to_file "$backup_path/backup_manifest", "\n";
+}
+
+# Create a file that can't be opened. (This is skipped on Windows.)
+sub mutilate_open_file_fails
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/PG_VERSION";
+	chmod(0, $pathname) || die "chmod $pathname: $!";
+}
+
+# Create a directory that can't be opened. (This is skipped on Windows.)
+sub mutilate_open_directory_fails
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/pg_subtrans";
+	chmod(0, $pathname) || die "chmod $pathname: $!";
+}
+
+# Create a directory that can't be searched. (This is skipped on Windows.)
+sub mutilate_search_directory_fails
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/base";
+	chmod(0400, $pathname) || die "chmod $pathname: $!";
+}
+
+# rmtree can't cope with a mode 400 directory, so change back to 700.
+sub cleanup_search_directory_fails 
+{
+	my ($backup_path) = @_;
+	my $pathname = "$backup_path/base";
+	chmod(0700, $pathname) || die "chmod $pathname: $!";
+}

src/bin/pg_validatebackup/t/004_options.pl

+# Verify the behavior of assorted pg_validatebackup options.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 25;
+
+# Start up the server and take a backup.
+my $master = get_new_node('master');
+$master->init(allows_streaming => 1);
+$master->start;
+my $backup_path = $master->backup_dir . '/test_options';
+$master->command_ok(['pg_basebackup', '-D', $backup_path, '--no-sync' ],
+					"base backup ok");
+
+# Verify that pg_validatebackup -q succeeds and produces no output.
+my $stdout;
+my $stderr;
+my $result = IPC::Run::run ['pg_validatebackup', '-q', $backup_path ],
+	'>', \$stdout, '2>', \$stderr;
+ok($result, "-q succeeds: exit code 0");
+is($stdout, '', "-q succeeds: no stdout");
+is($stderr, '', "-q succeeds: no stderr");
+
+# Corrupt the PG_VERSION file.
+my $version_pathname = "$backup_path/PG_VERSION";
+my $version_contents = slurp_file($version_pathname);
+open(my $fh, '>', $version_pathname) || die "open $version_pathname: $!";
+print $fh 'q' x length($version_contents);
+close($fh);
+
+# Verify that pg_validatebackup -q now fails.
+command_fails_like(['pg_validatebackup', '-q', $backup_path ],
+				   qr/checksum mismatch for file \"PG_VERSION\"/,
+				   '-q checksum mismatch');
+
+# Since we didn't change the length of the file, validation should succeed
+# if we ignore checksums. Check that we get the right message, too.
+command_like(['pg_validatebackup', '-s', $backup_path ],
+			 qr/backup successfully verified/,
+			 '-s skips checksumming');
+
+# Validation should succeed if we ignore the problem file.
+command_like(['pg_validatebackup', '-i', 'PG_VERSION', $backup_path ],
+			 qr/backup successfully verified/,
+			 '-i ignores problem file');
+
+# PG_VERSION is already corrupt; let's try also removing all of pg_xact.
+rmtree($backup_path . "/pg_xact");
+
+# We're ignoring the problem with PG_VERSION, but not the problem with
+# pg_xact, so validation should fail here.
+command_fails_like(['pg_validatebackup', '-i', 'PG_VERSION', $backup_path ],
+				   qr/pg_xact.*is present in the manifest but not on disk/,
+				   '-i does not ignore all problems');
+
+# If we use -i twice, we should be able to ignore all of the problems.
+command_like(['pg_validatebackup', '-i', 'PG_VERSION', '-i', 'pg_xact',
+				$backup_path ],
+			 qr/backup successfully verified/,
+			 'multiple -i options work');
+
+# Verify that when -i is not used, both problems are reported.
+$result = IPC::Run::run ['pg_validatebackup', $backup_path ],
+	'>', \$stdout, '2>', \$stderr;
+ok(!$result, "multiple problems: fails");
+like($stderr, qr/pg_xact.*is present in the manifest but not on disk/,
+	 "multiple problems: missing files reported");
+like($stderr, qr/checksum mismatch for file \"PG_VERSION\"/,
+	 "multiple problems: checksum mismatch reported");
+
+# Verify that when -e is used, only the problem detected first is reported.
+$result = IPC::Run::run ['pg_validatebackup', '-e', $backup_path ],
+	'>', \$stdout, '2>', \$stderr;
+ok(!$result, "-e reports 1 error: fails");
+like($stderr, qr/pg_xact.*is present in the manifest but not on disk/,
+	 "-e reports 1 error: missing files reported");
+unlike($stderr, qr/checksum mismatch for file \"PG_VERSION\"/,
+	   "-e reports 1 error: checksum mismatch not reported");
+
+# Test valid manifest with nonexistent backup directory.
+command_fails_like(['pg_validatebackup', '-m', "$backup_path/backup_manifest",
+						"$backup_path/fake" ],
+				   qr/could not open directory/,
+				   'nonexistent backup directory');

src/bin/pg_validatebackup/t/005_bad_manifest.pl

+# Test the behavior of pg_validatebackup when the backup manifest has
+# problems.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 58;
+
+my $tempdir = TestLib::tempdir;
+
+test_bad_manifest('input string ended unexpectedly',
+	  qr/could not parse backup manifest: The input string ended unexpectedly/,
+	  <<EOM);
+{
+EOM
+
+test_parse_error('unexpected object end', <<EOM);
+{}
+EOM
+
+test_parse_error('unexpected array start', <<EOM);
+[]
+EOM
+
+test_parse_error('expected version indicator', <<EOM);
+{"not-expected": 1}
+EOM
+
+test_parse_error('unexpected manifest version', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": "phooey"}
+EOM
+
+test_parse_error('unexpected scalar', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": true}
+EOM
+
+test_parse_error('unknown toplevel field', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Oops": 1}
+EOM
+
+test_parse_error('unexpected object start', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": {}}
+EOM
+
+test_parse_error('missing pathname', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [{}]}
+EOM
+
+test_parse_error('both pathname and encoded pathname', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Encoded-Path": "1234"}
+]}
+EOM
+
+test_parse_error('unexpected file field', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Oops": 1}
+]}
+EOM
+
+test_parse_error('missing size', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x"}
+]}
+EOM
+
+test_parse_error('file size is not an integer', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Size": "Oops"}
+]}
+EOM
+
+test_parse_error('unable to decode filename', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Encoded-Path": "123", "Size": 0}
+]}
+EOM
+
+test_fatal_error('duplicate pathname in backup manifest', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Size": 0},
+    {"Path": "x", "Size": 0}
+]}
+EOM
+
+test_parse_error('checksum without algorithm', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Size": 100, "Checksum": "Oops"}
+]}
+EOM
+
+test_fatal_error('unrecognized checksum algorithm', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Size": 100, "Checksum-Algorithm": "Oops", "Checksum": "00"}
+]}
+EOM
+
+test_fatal_error('invalid checksum for file', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [
+    {"Path": "x", "Size": 100, "Checksum-Algorithm": "CRC32C", "Checksum": "0"}
+]}
+EOM
+
+test_parse_error('missing start LSN', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": 1}
+]}
+EOM
+
+test_parse_error('missing end LSN', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": 1, "Start-LSN": "0/0"}
+]}
+EOM
+
+test_parse_error('unexpected wal range field', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Oops": 1}
+]}
+EOM
+
+test_parse_error('missing timeline', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {}
+]}
+EOM
+
+test_parse_error('unexpected object end', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": 1, "Start-LSN": "0/0", "End-LSN": "0/0"}
+]}
+EOM
+
+test_parse_error('timeline is not an integer', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": true, "Start-LSN": "0/0", "End-LSN": "0/0"}
+]}
+EOM
+
+test_parse_error('unable to parse start LSN', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": 1, "Start-LSN": "oops", "End-LSN": "0/0"}
+]}
+EOM
+
+test_parse_error('unable to parse end LSN', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "WAL-Ranges": [
+    {"Timeline": 1, "Start-LSN": "0/0", "End-LSN": "oops"}
+]}
+EOM
+
+test_parse_error('expected at least 2 lines', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [], "Manifest-Checksum": null}
+EOM
+
+my $manifest_without_newline = <<EOM;
+{"PostgreSQL-Backup-Manifest-Version": 1,
+ "Files": [],
+ "Manifest-Checksum": null}
+EOM
+chomp($manifest_without_newline);
+test_parse_error('last line not newline-terminated',
+				 $manifest_without_newline);
+
+test_fatal_error('invalid manifest checksum', <<EOM);
+{"PostgreSQL-Backup-Manifest-Version": 1, "Files": [],
+ "Manifest-Checksum": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234567890-"}
+EOM
+
+sub test_parse_error
+{
+	my ($test_name, $manifest_contents) = @_;
+
+	test_bad_manifest($test_name,
+					  qr/could not parse backup manifest: $test_name/,
+					  $manifest_contents);
+}
+
+sub test_fatal_error
+{
+	my ($test_name, $manifest_contents) = @_;
+
+	test_bad_manifest($test_name,
+					  qr/fatal: $test_name/,
+					  $manifest_contents);
+}
+
+sub test_bad_manifest
+{
+	my ($test_name, $regexp, $manifest_contents) = @_;
+
+	open(my $fh, '>', "$tempdir/backup_manifest") || die "open: $!";
+	print $fh $manifest_contents;
+	close($fh);
+
+	command_fails_like(['pg_validatebackup', $tempdir], $regexp,
+					   $test_name);
+}

src/bin/pg_validatebackup/t/006_encoding.pl

+# Verify that pg_validatebackup handles hex-encoded filenames correctly.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 5;
+
+my $master = get_new_node('master');
+$master->init(allows_streaming => 1);
+$master->start;
+my $backup_path = $master->backup_dir . '/test_encoding';
+$master->command_ok(['pg_basebackup', '-D', $backup_path, '--no-sync',
+					'--manifest-force-encode' ],
+					"backup ok with forced hex encoding");
+
+my $manifest = slurp_file("$backup_path/backup_manifest");
+my $count_of_encoded_path_in_manifest =
+	(() = $manifest =~ /Encoded-Path/mig);
+cmp_ok($count_of_encoded_path_in_manifest, '>', 100,
+	   "many paths are encoded in the manifest");
+
+command_like(['pg_validatebackup', '-s', $backup_path ],
+			 qr/backup successfully verified/,
+			 'backup with forced encoding validated');

src/bin/pg_validatebackup/t/007_wal.pl

+# Test pg_validatebackup's WAL validation.
+
+use strict;
+use warnings;
+use Cwd;
+use Config;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# Start up the server and take a backup.
+my $master = get_new_node('master');
+$master->init(allows_streaming => 1);
+$master->start;
+my $backup_path = $master->backup_dir . '/test_wal';
+$master->command_ok(['pg_basebackup', '-D', $backup_path, '--no-sync' ],
+					"base backup ok");
+
+# Rename pg_wal.
+my $original_pg_wal = $backup_path . '/pg_wal';
+my $relocated_pg_wal = $master->backup_dir . '/relocated_pg_wal';
+rename($original_pg_wal, $relocated_pg_wal) || die "rename pg_wal: $!";
+
+# WAL validation should fail.
+command_fails_like(['pg_validatebackup', $backup_path ],
+				   qr/WAL parsing failed for timeline 1/,
+				   'missing pg_wal causes failure');
+
+# Should work if we skip WAL verification.
+command_ok(['pg_validatebackup', '-n', $backup_path ],
+		   'missing pg_wal OK if not verifying WAL');
+
+# Should also work if we specify the correct WAL location.
+command_ok(['pg_validatebackup', '-w', $relocated_pg_wal, $backup_path ],
+		   '-w can be used to specify WAL directory');
+
+# Move directory back to original location.
+rename($relocated_pg_wal, $original_pg_wal) || die "rename pg_wal back: $!";
+
+# Get a list of files in that directory that look like WAL files.
+my @walfiles = grep { /^[0-9A-F]{24}$/ } slurp_dir($original_pg_wal);
+
+# Replace the contents of one of the files with garbage of equal length.
+my $wal_corruption_target = $original_pg_wal . '/' . $walfiles[0];
+my $wal_size = -s $wal_corruption_target;
+open(my $fh, '>', $wal_corruption_target)
+	|| die "open $wal_corruption_target: $!";
+print $fh 'w' x $wal_size;
+close($fh);
+
+# WAL validation should fail.
+command_fails_like(['pg_validatebackup', $backup_path ],
+				   qr/WAL parsing failed for timeline 1/,
+				   'corrupt WAL file causes failure');

src/include/replication/basebackup.h

@@ -12,6 +12,7 @@
 #ifndef _BASEBACKUP_H
 #define _BASEBACKUP_H
 
+#include "lib/stringinfo.h"
 #include "nodes/replnodes.h"
 
 /*
@@ -29,8 +30,12 @@ typedef struct
 	int64		size;
 } tablespaceinfo;
 
+struct manifest_info;
+typedef struct manifest_info manifest_info;
+
 extern void SendBaseBackup(BaseBackupCmd *cmd);
 
-extern int64 sendTablespace(char *path, bool sizeonly);
+extern int64 sendTablespace(char *path, char *oid, bool sizeonly,
+							manifest_info *manifest);
 
 #endif							/* _BASEBACKUP_H */

src/include/replication/walsender.h

@@ -38,6 +38,7 @@ extern bool log_replication_commands;
 extern void InitWalSender(void);
 extern bool exec_replication_command(const char *query_string);
 extern void WalSndErrorCleanup(void);
+extern void WalSndResourceCleanup(bool isCommit);
 extern void WalSndSignals(void);
 extern Size WalSndShmemSize(void);
 extern void WalSndShmemInit(void);