Add Generic WAL interface

This interface is designed to give an access to WAL for extensions which
could implement new access method, for example. Previously it was
impossible because restoring from custom WAL would need to access system
catalog to find a redo custom function. This patch suggests generic way
to describe changes on page with standart layout.

Bump XLOG_PAGE_MAGIC because of new record type.

Author: Alexander Korotkov with a help of Petr Jelinek, Markus Nullmeier and
	minor editorization by my
Reviewers: Petr Jelinek, Alvaro Herrera, Teodor Sigaev, Jim Nasby,
	Michael Paquier

doc/src/sgml/filelist.sgml

@@ -100,6 +100,7 @@
 <!ENTITY sources    SYSTEM "sources.sgml">
 <!ENTITY storage    SYSTEM "storage.sgml">
 <!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
+<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
 
 <!-- contrib information -->
 <!ENTITY contrib         SYSTEM "contrib.sgml">

doc/src/sgml/generic-wal.sgml

+<!-- doc/src/sgml/generic-wal.sgml -->
+
+<chapter id="generic-wal">
+ <title>Generic WAL records</title>
+
+  <para>
+   Despite all built-in access methods and WAL-logged modules having their own
+   types of WAL records, there is also a generic WAL record type, which describes
+   changes to pages in a generic way.  This is useful for extensions that
+   provide custom access methods, because they cannot register their own
+   WAL redo routines.
+  </para>
+
+  <para>
+   The API for contructing generic WAL records is defined in
+   <filename>generic_xlog.h</> and implemented in <filename>generic_xlog.c</>.
+   Each generic WAL record must be constructed by following these steps:
+
+   <orderedlist>
+    <listitem>
+     <para>
+      <function>state = GenericXLogStart(relation)</> &mdash; start
+      construction of a generic xlog record for the given relation.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      <function>page = GenericXLogRegister(state, buffer, isNew)</> &mdash;
+      register one or more buffers (one at a time) for the current generic
+      xlog record.  This function returns a copy of the page image, where
+      modifications can be made.  The second argument indicates if the page
+      is new (eventually, this will result in a full page image being put into
+      the xlog record).
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      Apply modifications to page images obtained in the previous step.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      <function>GenericXLogAbort(state)</> &mdash; finish construction of
+      a generic xlog record.
+     </para>
+    </listitem>
+   </orderedlist>
+  </para>
+
+  <para>
+   The xlog record construction can be canceled between any of the above
+   steps by calling <function>GenericXLogAbort()</>.  This will discard all
+   changes to the page image copies.
+  </para>
+
+  <para>
+   Please note the following points when constructing generic xlog records:
+   <itemizedlist>
+    <listitem>
+     <para>
+      No direct modifications of page images are allowed!  All modifications
+      must be done in copies acquired from <function>GenericXLogRegister()</>.
+      In other words, code which makes generic xlog records must never call
+      <function>BufferGetPage()</>.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      Registrations of buffers (step 2) and modifications of page images
+      (step 3) can be mixed freely, i.e., both steps may be repeated in any
+      sequence.  The only restriction is that you can modify a page image
+      only after the registration of the corresponding buffer.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      After registration, the buffer can also be unregistered by calling
+      <function>GenericXLogUnregister(buffer)</>.  In this case, the changes
+      made to that particular page image copy will be discarded.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      Generic xlog assumes that pages are using standard layout.  I.e., all
+      information between pd_lower and pd_upper will be discarded.
+     </para>
+    </listitem>
+
+    <listitem>
+     <para>
+      The maximum number of buffers that can be simultaneously registered
+      for a generic xlog is <literal>MAX_GENERIC_XLOG_PAGES</>.  An error will
+      be thrown if this limit is exceeded.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      Since you modify copies of page images, <function>GenericXLogStart()</>
+      does not start a critical section.  Thus, you can do memory allocation,
+      error throwing, etc. between <function>GenericXLogStart()</> and
+      <function>GenericXLogFinish()</>.  The actual critical section is present
+      inside <function>GenericXLogFinish()</>.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      <function>GenericXLogFinish()</> takes care of marking buffers as dirty
+      and setting their LSNs.  You do not need to do this explicitly.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      For unlogged relations, everything works the same except there is no
+      WAL record produced.  Thus, you typically do not need to do any explicit
+      checks for unlogged relations.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      If a registered buffer is not new, the generic xlog record contains
+      a delta between the old and the new page images.  This delta is produced
+      using per byte comparison.  The current delta mechanism is not effective
+      for moving data within a page and may be improved in the future.
+     </para>
+    </listitem>
+    <listitem>
+     <para>
+      The generic xlog redo function will acquire exclusive locks to buffers
+      in the same order as they were registered.  After redoing all changes,
+      the locks will be released in the same order.
+     </para>
+    </listitem>
+   </itemizedlist>
+  </para>
+</chapter>

doc/src/sgml/postgres.sgml

@@ -247,6 +247,7 @@
   &custom-scan;
   &geqo;
   &indexam;
+  &generic-wal;
   &gist;
   &spgist;
   &gin;

src/backend/access/rmgrdesc/Makefile

@@ -8,9 +8,9 @@ subdir = src/backend/access/rmgrdesc
 top_builddir = ../../../..
 include $(top_builddir)/src/Makefile.global
 
-OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o gindesc.o gistdesc.o \
-	   hashdesc.o heapdesc.o mxactdesc.o nbtdesc.o relmapdesc.o \
-	   replorigindesc.o seqdesc.o smgrdesc.o spgdesc.o \
+OBJS = brindesc.o clogdesc.o committsdesc.o dbasedesc.o genericdesc.o \
+	   gindesc.o gistdesc.o hashdesc.o heapdesc.o mxactdesc.o nbtdesc.o \
+	   relmapdesc.o replorigindesc.o seqdesc.o smgrdesc.o spgdesc.o \
 	   standbydesc.o tblspcdesc.o xactdesc.o xlogdesc.o
 
 include $(top_srcdir)/src/backend/common.mk

src/backend/access/rmgrdesc/genericdesc.c

+/*-------------------------------------------------------------------------
+ *
+ * genericdesc.c
+ *	  rmgr descriptor routines for access/transam/generic_xlog.c
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/access/rmgrdesc/genericdesc.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/generic_xlog.h"
+#include "lib/stringinfo.h"
+#include "storage/relfilenode.h"
+
+/*
+ * Description of generic xlog record: write page regions that this record
+ * overrides.
+ */
+void
+generic_desc(StringInfo buf, XLogReaderState *record)
+{
+	Pointer		ptr = XLogRecGetData(record),
+				end = ptr + XLogRecGetDataLen(record);
+
+	while (ptr < end)
+	{
+		OffsetNumber	offset,
+						length;
+
+		memcpy(&offset, ptr, sizeof(offset));
+		ptr += sizeof(offset);
+		memcpy(&length, ptr, sizeof(length));
+		ptr += sizeof(length);
+		ptr += length;
+
+		if (ptr < end)
+			appendStringInfo(buf, "offset %u, length %u; ", offset, length);
+		else
+			appendStringInfo(buf, "offset %u, length %u", offset, length);
+	}
+
+	return;
+}
+
+/*
+ * Identification of generic xlog record: we don't distinguish any subtypes
+ * inside generic xlog records.
+ */
+const char *
+generic_identify(uint8 info)
+{
+	return "Generic";
+}

src/backend/access/transam/Makefile

@@ -12,8 +12,8 @@ subdir = src/backend/access/transam
 top_builddir = ../../../..
 include $(top_builddir)/src/Makefile.global
 
-OBJS = clog.o commit_ts.o multixact.o parallel.o rmgr.o slru.o subtrans.o \
-	timeline.o transam.o twophase.o twophase_rmgr.o varsup.o \
+OBJS = clog.o commit_ts.o generic_xlog.o multixact.o parallel.o rmgr.o slru.o \
+	subtrans.o timeline.o transam.o twophase.o twophase_rmgr.o varsup.o \
 	xact.o xlog.o xlogarchive.o xlogfuncs.o \
 	xloginsert.o xlogreader.o xlogutils.o
 

src/backend/access/transam/rmgr.c

@@ -11,6 +11,7 @@
 #include "access/commit_ts.h"
 #include "access/gin.h"
 #include "access/gist_private.h"
+#include "access/generic_xlog.h"
 #include "access/hash.h"
 #include "access/heapam_xlog.h"
 #include "access/brin_xlog.h"

src/backend/replication/logical/decode.c

@@ -143,6 +143,7 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
 		case RM_BRIN_ID:
 		case RM_COMMIT_TS_ID:
 		case RM_REPLORIGIN_ID:
+		case RM_GENERIC_ID:
 			/* just deal with xid, and done */
 			ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(record),
 									buf.origptr);

src/bin/pg_xlogdump/.gitignore

@@ -4,6 +4,7 @@
 /clogdesc.c
 /committsdesc.c
 /dbasedesc.c
+/genericdesc.c
 /gindesc.c
 /gistdesc.c
 /hashdesc.c

src/bin/pg_xlogdump/rmgrdesc.c

@@ -11,6 +11,7 @@
 #include "access/brin_xlog.h"
 #include "access/clog.h"
 #include "access/commit_ts.h"
+#include "access/generic_xlog.h"
 #include "access/gin.h"
 #include "access/gist_private.h"
 #include "access/hash.h"

src/include/access/generic_xlog.h

+/*-------------------------------------------------------------------------
+ *
+ * generic_xlog.h
+ *	  Generic xlog API definition.
+ *
+ *
+ * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/generic_xlog.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef GENERIC_XLOG_H
+#define GENERIC_XLOG_H
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xloginsert.h"
+#include "storage/bufpage.h"
+#include "utils/rel.h"
+
+#define MAX_GENERIC_XLOG_PAGES	XLR_NORMAL_MAX_BLOCK_ID
+
+/* state of generic xlog record construction */
+struct GenericXLogState;
+typedef struct GenericXLogState GenericXLogState;
+
+/* API for construction of generic xlog records */
+extern GenericXLogState *GenericXLogStart(Relation relation);
+extern Page GenericXLogRegister(GenericXLogState *state, Buffer buffer,
+								bool isNew);
+extern void GenericXLogUnregister(GenericXLogState *state, Buffer buffer);
+extern XLogRecPtr GenericXLogFinish(GenericXLogState *state);
+extern void GenericXLogAbort(GenericXLogState *state);
+
+/* functions defined for rmgr */
+extern void generic_redo(XLogReaderState *record);
+extern const char *generic_identify(uint8 info);
+extern void generic_desc(StringInfo buf, XLogReaderState *record);
+
+#endif   /* GENERIC_XLOG_H */

src/include/access/rmgrlist.h

@@ -45,3 +45,4 @@ PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_start
 PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL)
 PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL)
 PG_RMGR(RM_REPLORIGIN_ID, "ReplicationOrigin", replorigin_redo, replorigin_desc, replorigin_identify, NULL, NULL)
+PG_RMGR(RM_GENERIC_ID, "Generic", generic_redo, generic_desc, generic_identify, NULL, NULL)

src/include/access/xlog_internal.h

@@ -31,7 +31,7 @@
 /*
  * Each page of XLOG file has a header like this:
  */
-#define XLOG_PAGE_MAGIC 0xD089	/* can be used as WAL version indicator */
+#define XLOG_PAGE_MAGIC 0xD090	/* can be used as WAL version indicator */
 
 typedef struct XLogPageHeaderData
 {