Prevent incorrect updates of pg_index while reindexing pg_index itself.

The places that attempt to change pg_index.indcheckxmin during a reindexing
operation cannot be executed safely if pg_index itself is the subject of
the operation.  This is the explanation for a couple of recent reports of
VACUUM FULL failing with
	ERROR:  duplicate key value violates unique constraint "pg_index_indexrelid_index"
	DETAIL:  Key (indexrelid)=(2678) already exists.

However, there isn't any real need to update indcheckxmin in such a
situation, if we assume that pg_index can never contain a truly broken HOT
chain.  This assumption holds if new indexes are never created on it during
concurrent operations, which is something we don't consider safe for any
system catalog, not just pg_index.  Accordingly, modify the code to not
manipulate indcheckxmin when reindexing any system catalog.

Back-patch to 8.3, where HOT was introduced.  The known failure scenarios
involve 9.0-style VACUUM FULL, so there might not be any real risk before
9.0, but let's not assume that.

src/backend/catalog/index.c

@@ -1770,6 +1770,13 @@ index_build(Relation heapRelation,
 		HeapTuple	indexTuple;
 		Form_pg_index indexForm;
 
+		/*
+		 * Broken HOT chains should not get reported in system catalogs; in
+		 * particular it would be quite dangerous to try to modify the index's
+		 * pg_index entry if we are reindexing pg_index itself.
+		 */
+		Assert(!IsSystemRelation(heapRelation));
+
 		pg_index = heap_open(IndexRelationId, RowExclusiveLock);
 
 		indexTuple = SearchSysCacheCopy1(INDEXRELID,
@@ -1827,7 +1834,13 @@ index_build(Relation heapRelation,
  * A side effect is to set indexInfo->ii_BrokenHotChain to true if we detect
  * any potentially broken HOT chains.  Currently, we set this if there are
  * any RECENTLY_DEAD entries in a HOT chain, without trying very hard to
- * detect whether they're really incompatible with the chain tip.
+ * detect whether they're really incompatible with the chain tip.  However,
+ * we do not ever set ii_BrokenHotChain true when the relation is a system
+ * catalog.  This is to avoid problematic behavior when reindexing pg_index
+ * itself: we can't safely change the index's indcheckxmin field when we're
+ * partway through such an operation.  It should be okay since the set of
+ * indexes on a system catalog ought not change during concurrent operations,
+ * so that no HOT chain in it could ever become broken.
  */
 double
 IndexBuildHeapScan(Relation heapRelation,
@@ -2004,7 +2017,8 @@ IndexBuildHeapScan(Relation heapRelation,
 					{
 						indexIt = false;
 						/* mark the index as unsafe for old snapshots */
-						indexInfo->ii_BrokenHotChain = true;
+						if (!is_system_catalog)
+							indexInfo->ii_BrokenHotChain = true;
 					}
 					else if (indexInfo->ii_BrokenHotChain)
 						indexIt = false;
@@ -2092,7 +2106,8 @@ IndexBuildHeapScan(Relation heapRelation,
 					{
 						indexIt = false;
 						/* mark the index as unsafe for old snapshots */
-						indexInfo->ii_BrokenHotChain = true;
+						if (!is_system_catalog)
+							indexInfo->ii_BrokenHotChain = true;
 					}
 					else if (indexInfo->ii_BrokenHotChain)
 						indexIt = false;
@@ -2787,8 +2802,13 @@ reindex_index(Oid indexId, bool skip_constraint_checks)
 	 * We can also reset indcheckxmin, because we have now done a
 	 * non-concurrent index build, *except* in the case where index_build
 	 * found some still-broken HOT chains.
+	 *
+	 * When reindexing a system catalog, don't do any of this --- it would be
+	 * particularly risky to try to modify pg_index while we are reindexing
+	 * pg_index itself.  We don't support CREATE INDEX CONCURRENTLY on system
+	 * catalogs anyway, and they should never have indcheckxmin set either.
 	 */
-	if (!skipped_constraint)
+	if (!skipped_constraint && !IsSystemRelation(heapRelation))
 	{
 		pg_index = heap_open(IndexRelationId, RowExclusiveLock);