The normalization-check target needs to be run last, after moving the
newly generated files into place.  Also, we need an additional
dependency so that unicode_norm.o is rebuilt first.  Otherwise,
norm_test will still test the old files but against the new expected
results, which will probably fail.

Jonathan Katz felt that slightly different indentation settings made
for a better-looking result, so sync stylesheet-fo.xsl (for PDF) and
stylesheet.css (for non-website-style HTML) with those choices.

Discussion: https://postgr.es/m/31464.1587156281@sss.pgh.pa.us

We have repeatedly seen the buildfarm reach the Assert(false) in
SyncRepGetSyncStandbysPriority.  This apparently is due to failing to
consider the possibility that the sync_standby_priority values in
shared memory might be inconsistent; but they will be whenever only
some of the walsenders have updated their values after a change in
the synchronous_standby_names setting.  That function is vastly too
complex for what it does, anyway, so rewriting it seems better than
trying to apply a band-aid fix.

Furthermore, the API of SyncRepGetSyncStandbys is broken by design:
it returns a list of WalSnd array indexes, but there is nothing
guaranteeing that the contents of the WalSnd array remain stable.
Thus, if some walsender exits and then a new walsender process
takes over that WalSnd array slot, a caller might make use of
WAL position data that it should not, potentially leading to
incorrect decisions about whether to release transactions that
are waiting for synchronous commit.

To fix, replace SyncRepGetSyncStandbys with a new function
SyncRepGetCandidateStandbys that copies all the required data
from shared memory while holding the relevant mutexes.  If the
associated walsender process then exits, this data is still safe to
make release decisions with, since we know that that much WAL *was*
sent to a valid standby server.  This incidentally means that we no
longer need to treat sync_standby_priority as protected by the
SyncRepLock rather than the per-walsender mutex.

SyncRepGetSyncStandbys is no longer used by the core code, so remove
it entirely in HEAD.  However, it seems possible that external code is
relying on that function, so do not remove it from the back branches.
Instead, just remove the known-incorrect Assert.  When the bug occurs,
the function will return a too-short list, which callers should treat
as meaning there are not enough sync standbys, which seems like a
reasonably safe fallback until the inconsistent state is resolved.
Moreover it's bug-compatible with what has been happening in non-assert
builds.  We cannot do anything about the walsender-replacement race
condition without an API/ABI break.

The bogus assertion exists back to 9.6, but 9.6 is sufficiently
different from the later branches that the patch doesn't apply at all.
I chose to just remove the bogus assertion in 9.6, feeling that the
probability of a bad outcome from the walsender-replacement race
condition is too low to justify rewriting the whole patch for 9.6.

Discussion: https://postgr.es/m/21519.1585272409@sss.pgh.pa.us

In some corner cases, this could also lead to corrupted values being
included in the tuple.

Users who are concerned that they are affected by this should first
upgrade and then perform a base backup of their database and restore onto
an off-line server. They should then query each table with generated
columns to ensure there are no rows where the generated expression does
not match a newly calculated version of the GENERATED ALWAYS expression.
If no crashes occur and no rows are returned then you're not affected.

Fixes bug #16369.

Reported-by: Cameron Ezell
Discussion: https://postgr.es/m/16369-5845a6f1bef59884@postgresql.org
Backpatch-through: 12 (where GENERATED ALWAYS columns were added.)

The table layout ideas proposed in commit e894c618 were not as widely
popular as I'd hoped.  After discussion, we've settled on a layout
that's effectively a single-column table with cell contents much like a
<varlistentry> description of the function or operator; though we're not
actually using <varlistentry>, because it'd add way too much vertical
space.  Instead the effect is accomplished using line-break processing
instructions to separate the description and example(s), plus CSS or FO
customizations to produce indentation of all but the first line in each
cell.  While technically this is a bit grotty, it does have the
advantage that we won't need to write nearly as much boilerplate markup.

This patch updates tables 9.30, 9.31, and 9.33 (which were touched by
the previous patch) to the revised style, and additionally converts
table 9.10.  A lot of work still remains to do, but hopefully it won't
be too controversial.

Thanks to Andrew Dunstan, Pierre Giraud, Robert Haas, Alvaro Herrera,
David Johnston, Jonathan Katz, Isaac Morland for valuable ideas.

Discussion: https://postgr.es/m/8691.1586798003@sss.pgh.pa.us

This reverts commit 9e24109f.

This caused build errors when building without openssl, and it's
simplest just to revert it.

This should have been protected by #ifdef USE_OPENSSL in commit
896fcdb2.

Per the real complaint this time from Daniel Gustafsson.

Apparently in some language versions of Visual Studio nmake outputs some
material after the version number and before the end of the line. This
has been seen in Chinese versions. Therefore, we no longer demand that
the version string comes at the end of a line.

Per complaint from Cuiping Lin.

Backpatch to all live branches.

In commit 4dc63552 I neglected to put #ifdef USE_OPENSSL around the
declarations of the new items. This is remedied here.

Per complaint from Daniel Gustafsson.

recordExtObjInitPriv and removeExtObjInitPriv were sloppy about
calling ReleaseSysCache.  The cases cannot occur given current usage
in ALTER EXTENSION ADD/DROP, since we wouldn't get here for these
relkinds; but it seems wise to clean up better.

In passing, extend test logic in test_pg_dump to exercise the
dropped-column code paths here.

Since the case is unreachable at present, there seems no great
need to back-patch; hence fix HEAD only.

Kyotaro Horiguchi, with test case and comment adjustments by me

Discussion: https://postgr.es/m/20200417.151831.1153577605111650154.horikyota.ntt@gmail.com

Author: Fujii Masao
Reviewed-by: Robert Haas
Discussion: https://postgr.es/m/951743d0-fd7e-8e2b-d489-1368a58b7304@oss.nttdata.com

The result of the query used to retrieve the WAL segment size from the
backend was not getting freed in two code paths.  Both pg_basebackup and
pg_receivewal exit immediately if a failure happened on this query, so
this was not an actual problem, but it could be an issue if this code
gets used for other tools in different ways, be they future tools in
this code tree or external, existing, ones.

Oversight in commit fc49e24f, so backpatch down to 11.

Author: Jie Zhang
Discussion: https://postgr.es/m/970ad9508461469b9450b64027842331@G08CNEXMBPEKD06.g08.fujitsu.local
Backpatch-through: 11

It was previously thought that remove_useless_groupby_columns() needed to
keep track of which constraints the generated plan depended upon, however,
this is unnecessary. The confusion likely arose regarding this because of
check_functional_grouping(), which does need to track the dependency to
ensure VIEWs with columns which are functionally dependant on the GROUP BY
remain so. For remove_useless_groupby_columns(), cached plans will just
become invalidated when the primary key's underlying index is removed
through the normal relcache invalidation code.

Here we just remove the unneeded code which records the dependency and
updates the comments. The previous comments claimed that we could not use
UNIQUE constraints for the same optimization due to lack of a
pg_constraint record for NOT NULL constraints (which are required because
NULLs can be duplicated in a unique index). Since we don't actually need a
pg_constraint record to handle the invalidation, it looks like we could
add code to do this in the future. But not today.

We're not really fixing any bug in the code here, this fix is just to set
the record straight on UNIQUE constraints. This code was added back in
9.6, but due to lack of any bug, we'll not be backpatching this.

Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/CAApHDvrdYa=VhOoMe4ZZjZ-G4ALnD-xuAeUNCRTL+PYMVN8OnQ@mail.gmail.com

fixup_whole_row_references() did the wrong thing with a dropped column,
resulting in a commit-time warning about a cache reference leak.

I (tgl) added a test case exercising this, but back-patched the test
only as far as v10; the patch didn't apply cleanly to 9.6 and it
didn't seem worth the trouble to adapt it.  The bug is pretty old
though, so apply the code change all the way back.

Michael Luo, with cosmetic improvements by me

Discussion: https://postgr.es/m/BYAPR08MB5606D1453D7F50E2AF4D2FD29AD80@BYAPR08MB5606.namprd08.prod.outlook.com

Earlier we were inconsistent in allowing the usage of parallel and
full options.  Change it such that we disallow them only when they are
combined in a way that we don't support.

In passing, improve the comments in some of the existing tests of parallel
vacuum.

Reported-by: Tushar Ahuja
Author: Justin Pryzby, Amit Kapila
Reviewed-by: Sawada Masahiko, Michael Paquier, Mahendra Singh Thalor and
Amit Kapila
Discussion: https://postgr.es/m/58c8d171-e665-6fa3-a9d3-d9423b694dae%40enterprisedb.com

Since 0d8c9c12, pg_basebackup would generate an error if connected to a
backend version older than 12 where backup manifests are not supported.
Avoiding this error is possible by using the --no-manifest option.

This error handling could be confusing for some users, where patching a
backup script that interacts with multiple backend versions would cause
the addition of --no-manifest to potentially not generate a backup
manifest even for Postgres 13 and newer versions.  As we want to
encourage the use of backup manifests as much as possible, this commit
silently disables manifests where not supported, instead of generating
an error.

While on it, rework a bit the code to make it more consistent with the
surroundings when generating the BASE_BACKUP command.

Per discussion with Andres Freund, Stephen Frost, Robert Haas, Álvaro
Herrera, Kyotaro Horiguchi, Tom Lane, David Steele, and me.

Author: Michael Paquier
Discussion: https://postgr.es/m/20200410080910.GZ1606@paquier.xyz

Spotted during post-commit review of the nbtree deduplication commit
(commit 0d861bbb).

A query used to read default ACL information from the catalogs did not
free a set of PQExpBuffer.

Oversight in commit e2090d9d, so backpatch down to 9.6.

Author: Jie Zhang
Reviewed-by: Sawada Masahiko
Discussion: https://postgr.es/m/05bcbc5857f948efa0b451b85a48ae10@G08CNEXMBPEKD06.g08.fujitsu.local
Backpatch-through: 9.6

This commit prevents pg_basebackup from receiving backup_manifest file
when --no-manifest is specified. Previously, when pg_basebackup was
writing a tarfile to stdout, it tried to receive backup_manifest file even
when --no-manifest was specified, and reported an error.

Also remove unused -m option from pg_basebackup.

Also fix typo in BASE_BACKUP command documentation.

Author: Fujii Masao
Reviewed-by: Michael Paquier, Robert Haas
Discussion: https://postgr.es/m/01e3ed3a-8729-5aaa-ca84-e60e3ca59db8@oss.nttdata.com

A comment from the Berkeley days incorrectly claimed that the page
management code cares about the contents of the hole in the center of
the page (at least in the case of the left half of an nbtree page
split).  Commit 8fa30f90 added an addendum that stated that the
original comment was "probably obsolete".  It's definitely obsolete,
though, so remove the original comment plus the addendum.

Commit 913bbd88 overlooked that the result of coerce_to_target_type
might need collation fixups.  Per report from Andreas Joseph Krogh.

Discussion: https://postgr.es/m/VisenaEmail.72.37d08ec2b8cb8fb5.17179940cd3@tc7-visena

The consensus of the project appears to be that this provides little
benefit and is simply an annoyance.

Discussion: https://postgr.es/m/27481.1586618092@sss.pgh.pa.us

Back in commits 1df92eea, f884a968, and 592123ef I used some
hackish code to set the script search path, unaware despite decades of
perl that there was a completely standard way to do this. This patch
changes those cases to use the standard perl FindBin package.

Patch by me, at the request of Andres Freund. Reviewed by
Justin Pryzby, Erik Rijkers, Álvaro Herrera, and Andrew
Dunstan.

Discussion: http://postgr.es/m/20200327203225.hcm6ag4grwsiruea@alap3.anarazel.de

Nest the "update metapage as part of insert into root-like page" branch
inside the broader "insert into internal page" branch.  This improves
readability.

This fixes some comments and documentation new as of Postgres 13, and is
a follow-up of the work done in dd0f37ec.

Author: Justin Pryzby
Discussion: https://postgr.es/m/20200408165653.GF2228@telsasoft.com

Clearly it's not okay for nbtree to split a page that is the only page
on its level, and then find that it has to split the parent one level up
in turn.  There is simply no code to handle the split_only_page case in
the _bt_insertonpg() "newitem won't fit" branch (only the "newitem fits"
branch handles split_only_page).  Add a defensive assertion that will
fail if a split_only_page call to _bt_insertonpg() somehow ends up
splitting the target/parent page.

I (pgeoghegan) believe that we don't need split_only_page handling for
the "newitem won't fit" branch because anybody calling _bt_insertonpg()
like this would have to hold a lock on the same one and only child page.

Reported-by: Justin Pryzby
Author: Justin Pryzby, with few changes by me
Reviewed-by: Amit Kapila and Sawada Masahiko
Discussion: https://postgr.es/m/20200322021801.GB2563@telsasoft.com

It seems like a good idea for nbtree's retail insert code to be
absolutely consistent with nbtree's page split code for anything that
naturally requires equivalent handling.  Anything that concerns
inserting newitem (which is handled as part of the page split atomic
action when a page split is required) should work in exactly the same
way.  With that in mind, make _bt_insertonpg() handle 'cbuf' in a way
that matches _bt_split().

Per buildfarm member hoverfly, a slow walsender could make the test
fail.  Back-patch to v10, where the test was introduced.

Discussion: https://postgr.es/m/20200414013849.GA886648@rfd.leadboat.com

Now that warnings are enabled across the board, this code that tries to
print an undef variable emits one.  Silently printing the empty string
achieves the previous behavior.

Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Reviewed-by: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>
Discussion: https://postgr.es/m/E1jO1VT-0008Qk-TM@gemulon.postgresql.org

An nbtree split point can be thought of as a point between two adjoining
tuples from an imaginary version of the page being split that includes
the incoming/new item (in addition to the items that really are on the
page).  These adjoining tuples are called the lastleft and firstright
tuples.

The variables that represent split points contained a field called
firstright, which is an offset number of the first data item from the
original page that goes on the new right page.  The corresponding tuple
from origpage was usually the same thing as the actual firstright tuple,
but not always: the firstright tuple is sometimes the new/incoming item
instead.  This situation seems unnecessarily confusing.

Make things clearer by renaming the origpage offset returned by
_bt_findsplitloc() to "firstrightoff".  We now have a firstright tuple
and a firstrightoff offset number which are comparable to the
newitem/lastleft tuples and the newitemoff/lastleftoff offset numbers
respectively.  Also make sure that we are consistent about how we
describe nbtree page split point state.

Push the responsibility for dealing with pg_upgrade'd !heapkeyspace
indexes down to lower level code, relieving _bt_split() from dealing
with it directly.  This means that we always have a palloc'd left page
high key on the leaf level, no matter what.  This enables simplifying
some of the code (and code comments) within _bt_split().

Finally, restructure the page split code to make it clearer why suffix
truncation (which only takes place during leaf page splits) is
completely different to the first data item truncation that takes place
during internal page splits.  Tuples are marked as having fewer
attributes stored in both cases, and the firstright tuple is truncated
in both cases, so it's easy to imagine somebody missing the distinction.

This replaces a few occurrences of ugly code with a more clean and
idiomatic usage. The problem was highlighted by perlcritic, but we're
not enforcing the policy that led to the discovery.

Discussion: https://postgr.es/m/20200412074245.GB623763@rfd.leadboat.com

We've had a mixture of the warnings pragma, the -w switch on the shebang
line, and no warnings at all. This patch removes the -w swicth and add
the warnings pragma to all perl sources missing it. It raises the
severity of the TestingAndDebugging::RequireUseWarnings  perlcritic
policy to level 5, so that we catch any future violations.

Discussion: https://postgr.es/m/20200412074245.GB623763@rfd.leadboat.com

This makes it easier to do a web search for details of the policy that's
been violated, as well as displaying the name that might be needed for a
policy override.

Various perlcritic settings changes are being discussed, but this one
should be uncontroversial.

The previous commit missed an instance.

Noriyoshi Shinoda

Discussion: http://postgr.es/m/TU4PR8401MB115291AE850BA7CF1AEB2F0BEEDD0@TU4PR8401MB1152.NAMPRD84.PROD.OUTLOOK.COM

Reported-by: Justin Pryzby and Euler Taveira
Author: Justin Pryzby and Julien Rouhaud
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/CAB-hujrP8ZfUkvL5OYETipQwA=e3n7oqHFU=4ZLxWS_Cza3kQQ@mail.gmail.com

Move the file name to a format parameter to ease translatability.  Add
error code where missing.  Make the wording consistent.

We've long fought with the draconian space limitations of our
traditional table layout for describing SQL functions and operators.
This commit introduces a new approach, though so far I've only applied
it to a few of those tables.  The new way makes use of DocBook's support
for different layouts in different rows of a table, and allows the
descriptions and examples for a function or operator to run to several
lines without as much ugliness and wasted space as before.

The core layout concept is now

     Name              Signature
                      Description
                 Example     Example Result

so that a function or operator really has three table rows not one,
but we group them to look like one row by having the name column
have only one entry for all three rows.  (Actually, there could be
four or more rows if you wanted to have more than one example, which
is another thing that was painful before but works easily now.)
This is handled by a "morerows" annotation on the name entry, which
isn't perfect (notably, the toolchain is not smart enough to avoid
breaking these row groups across PDF pages) but there seems no better
solution in DocBook.  The name column is normally fairly narrow,
allowing plenty of space for the other column(s), and not wasting too
much space when one of the other components runs to multiple lines.

The varying row layout is managed by defining named "spans" and then
tagging entries with a "spanname" of "name", "sig", "desc", "example",
or "exresult".  This provides a bit of semantic annotation to go with
the formatting improvement, which seems like a good thing.  (It seems
that we have to re-define these spans afresh for each table, which is
annoying, but it's not any worse than the duplication involved in
the table headers.  At least that gives us an opportunity to vary the
relative column widths per-table, which is handy since function tables
sometimes need much wider name columns than operator tables.)

Signature entries should be written in the style
    <function>fname</function>(<type>typename</type> ...)
    <returnvalue>typename</returnvalue>
The <returnvalue> tag produces a right arrow before the result type
name.  (I'll document that convention in a user-visible place later.)

While this provides significantly more horizontal space than before
for examples, it's still true that PDF output is a lot narrower than
typical webpage viewing windows, so some examples need to be broken
in places where there is no whitespace.  I've added &zwsp; markers in
suitable places to allow the tables to render warning-free in PDF.

I've so far converted only the date/time operator, date/time function,
and enum function tables in sections 9.9 and 9.10; these were chosen
to provide a reasonable sample of the formatting problems that need
to be solved.  Assuming that this looks good on the website and doesn't
provoke howls of anguish, I'll work on the other similar tables in the
near future.

There's a moderate amount of new editorial content in this patch along
with the raw formatting changes; for instance I had to write text
descriptions for operators that lacked them.  I failed to resist the
temptation to improve some other descriptions and examples, too.

Patch by me, with thanks to Alexander Lakhin for assistance with
figuring out some formatting issues.

Discussion: https://postgr.es/m/9326.1581457869@sss.pgh.pa.us

We already had a couple of places using zero-width spaces for formatting
hackery, and we're going to need more if we ever want the PDF manuals to
look decent.  But please let's not write hard-coded Unicode escapes.
We can avoid that by using a custom entity, which also provides a place
to put a teeny bit of documentation about what it is and how to use it.

I'd previously posted a patch using "&break;" for this, but on reflection
that would be horrible to grep for.  Instead let's use "&zwsp;", based
on the name of the Unicode symbol ("zero width space").

Discussion: https://postgr.es/m/9326.1581457869@sss.pgh.pa.us