Mark the fields that should be accessed via partitioning-related
functions, as we already did for some other fields.

Amit Langote

Discussion: https://postgr.es/m/CA+HiwqFnK6LbVMACMCaqwWrvoSFTecZzufKRahg2qGvLPYMX=g@mail.gmail.com

Instead of passing around a pointer to the RangeTblEntry that
provides the desired column, pass a pointer to the associated
ParseNamespaceItem.  The RTE is trivially reachable from the nsitem,
and having the ParseNamespaceItem allows access to additional
information.  As proof of concept for that, add the rangetable index
to ParseNamespaceItem, and use that to get rid of RTERangeTablePosn
searches.

(I have in mind to teach the parser to generate some different
representation for Vars that are nullable by outer joins, and
keeping the necessary information in ParseNamespaceItems seems
like a reasonable approach to that.  But whether that ever
happens or not, this seems like good cleanup.)

Also refactor the code around scanRTEForColumn so that the
"fuzzy match" stuff does not leak out of parse_relation.c.

Discussion: https://postgr.es/m/26144.1576858373@sss.pgh.pa.us

confirmed_flush is part of a replication slot's information, but not
confirmed_lsn.

Author: Kyotaro Horiguchi
Discussion: https://postgr.es/m/20191226.175919.17237335658671970.horikyota.ntt@gmail.com
Backpatch-through: 11

The part in charge of doing the vacuum on all the indexes of a relation
was duplicated, with the same handling for progress reporting done.
While on it, update the progress reporting for heap vacuuming in the
subroutine doing the actual work, keeping the status update local.  This
way, any future caller of lazy_vacuum_heap() does not have to worry
about doing any progress reporting update.

Author: Justin Pryzby, Michael Paquier
Discussion: https://postgr.es/m/20191120210600.GC30362@telsasoft.com

Column defaults may be specified separately for each partition.
But INSERT via a partitioned table ignores those partition's default values.
The former is documented, but the latter restriction not.
This commit adds the note about that restriction into the document.

Author: Fujii Masao
Reviewed-by: Amit Langote
Discussion: https://postgr.es/m/CAHGQGwEs-59omrfGF7hOHz9iMME3RbKy5ny+iftDx3LHTEn9sA@mail.gmail.com

In the wake of commit 5b931237, there's no particular reason
for this restriction (previously, it was problematic because of
the implied rowtype reference).  A simple constraint on a whole-row
Var probably isn't that useful, but conceivably somebody would want
to pass one to a function that extracts a partitioning key.  Besides
which, we're expending much more code to enforce the restriction than
we save by having it, since the latter quantity is now zero.
So drop the restriction.

Amit Langote

Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com

This is dead code in the wake of the previous commit.
We can always add it back if we need it again someday.

Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com

Formerly the rd_partkey and rd_partdesc data structures were always
populated immediately when a relcache entry was built or rebuilt.
This patch changes things so that they are populated only when they
are first requested.  (Hence, callers *must* now always use
RelationGetPartitionKey or RelationGetPartitionDesc; just fetching
the pointer directly is no longer acceptable.)

This seems to have some performance benefits, but the main reason to do
it is that it eliminates a recursive-reload failure that occurs if the
partkey or partdesc expressions contain any references to the relation's
rowtype (as discovered by Amit Langote).  In retrospect, since loading
these data structures might result in execution of nearly-arbitrary code
via eval_const_expressions, it was a dumb idea to require that to happen
during relcache entry rebuild.

Also, fix things so that old copies of a relcache partition descriptor
will be dropped when the cache entry's refcount goes to zero.  In the
previous coding it was possible for such copies to survive for the
lifetime of the session, as I'd complained of in a previous discussion.
(This management technique still isn't perfect, but it's better than
before.)  Improve the commentary explaining how that works and why
it's safe to hand out direct pointers to these relcache substructures.

In passing, improve RelationBuildPartitionDesc by using the same
memory-context-parent-swap approach used by RelationBuildPartitionKey,
thereby making it less dependent on strong assumptions about what
partition_bounds_copy does.  Avoid doing get_rel_relkind in the
critical section, too.

Patch by Amit Langote and Tom Lane; Robert Haas deserves some credit
for prior work in the area, too.  Although this is a pre-existing
problem, no back-patch: the patch seems too invasive to be safe to
back-patch, and the bug it fixes is a corner case that seems
relatively unlikely to cause problems in the field.

Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com
Discussion: https://postgr.es/m/CA+TgmoY3bRmGB6-DUnoVy5fJoreiBJ43rwMrQRCdPXuKt4Ykaw@mail.gmail.com

The following renaming is done so as source files related to index
access methods are more consistent with table access methods (the
original names used for index AMs ware too generic, and could be
confused as including features related to table AMs):
- amapi.h -> indexam.h.
- amapi.c -> indexamapi.c.  Here we have an equivalent with
backend/access/table/tableamapi.c.
- amvalidate.c -> indexamvalidate.c.
- amvalidate.h -> indexamvalidate.h.
- genam.c -> indexgenam.c.
- genam.h -> indexgenam.h.

This has been discussed during the development of v12 when table AM was
worked on, but the renaming never happened.

Author: Michael Paquier
Reviewed-by: Fabien Coelho, Julien Rouhaud
Discussion: https://postgr.es/m/20191223053434.GF34339@paquier.xyz

Using \ is unnecessary and ugly, so remove that.  While at it, stitch
the literals back into a single line: we've long discouraged splitting
error message literals even when they go past the 80 chars line limit,
to improve greppability.

Leave contrib/tablefunc alone.

Discussion: https://postgr.es/m/20191223195156.GA12271@alvherre.pgsql

Since d6c55de1, src/port/snprintf.c is able to use %m instead of
strerror().  A couple of utilities in src/bin/ have already done the
switch, and do it now for pg_waldump as this reduces the workload for
translators.

Note that more could be done, particularly with pgbench.  Thanks to
Kyotaro Horiguchi for the discussion.

Discussion: https://postgr.es/m/20191129065115.GM2505@paquier.xyz

Our algorithm for choosing batch numbers turned out not to work
effectively for multi-billion key inner relations.  We would use
more hash bits than we have, and effectively concentrate all tuples
into a smaller number of batches than we intended.  While ideally
we should switch to wider hashes, for now, change the algorithm to
one that effectively gives up bits from the bucket number when we
don't have enough bits.  That means we'll finish up with longer
bucket chains than would be ideal, but that's better than having
batches that don't fit in work_mem and can't be divided.

Batch-patch to all supported releases.

Author: Thomas Munro
Reviewed-by: Tom Lane, thanks also to Tomas Vondra, Alvaro Herrera, Andres Freund for testing and discussion
Reported-by: James Coleman
Discussion: https://postgr.es/m/16104-dc11ed911f1ab9df%40postgresql.org

While building a hash map of categories in load_categories_hash,
resulting category names have not thus far been checked to ensure
they are not null. Prior to pg12 null category names worked to the
extent that they did not crash on some platforms. This is because
those system libraries have an snprintf which can deal with being
passed a null pointer argument for a string. But even in those cases
null categories did nothing useful. And on some platforms it crashed.
As of pg12, our own version of snprintf gets called, and it does
not deal with null pointer arguments at all, and crashes consistently.

Fix that by disallowing null categories. They never worked usefully,
and no one has ever asked for them to work previously. Back-patch to
all supported branches.

Reported-By: Ireneusz Pluta
Discussion: https://postgr.es/m/16176-7489719b05e4303c@postgresql.org

This wasn't checked originally, but it should have been, because
in general pseudo-types can't be stored to and retrieved from disk.
Notably, partition bound values of type "record" would not be
interpretable by another session.

In v12 and HEAD, add another flag to CheckAttributeType's repertoire
so that it can produce a specific error message for this case.  That's
infeasible in older branches without an ABI break, so fall back to
a slightly-less-nicely-worded error message in v10 and v11.

Problem noted by Amit Langote, though this patch is not his initial
solution.  Back-patch to v10 where partitioning was introduced.

Discussion: https://postgr.es/m/CA+HiwqFUzjfj9HEsJtYWcr1SgQ_=iCAvQ=O2Sx6aQxoDu4OiHw@mail.gmail.com

We probably should have thought of this case when ranges were added,
but we didn't.  (It's not the fault of commit eb51af71, because
ranges didn't exist then.)

It's an old bug, so back-patch to all supported branches.

Discussion: https://postgr.es/m/7782.1577051475@sss.pgh.pa.us

Use get_publication_oid() instead of reimplementing it.

Discussion: https://postgr.es/m/20191220201017.GA17292@alvherre.pgsql

Make a function prototype argument's name match the function
definition's argument name.

Comments about the consequences of clearing the BTP_HAS_GARBAGE page
flag bit that apply only to VACUUM were added to code that deals with
opportunistic deletion of LP_DEAD items by commit a760893d.  The same
comment block was added to both _bt_delitems_vacuum() and
_bt_delitems_delete().  Correct _bt_delitems_delete()'s copy of the
comment block.

_bt_delitems_delete() reliably deletes items that were found by caller
to have their LP_DEAD bit set.  There is no question about whether or
not unsetting the BTP_HAS_GARBAGE bit can miss some LP_DEAD items that
were set recently.

Also tweak a related section of the nbtree README.

If the first transaction block in these tests were entered exactly
at midnight (California time), they'd report a bogus failure due
to 'now' and 'midnight' having the same values.  Commit 8c2ac75c
had dismissed this as being of negligible probability, but we've
now seen it happen in the buildfarm, so let's prevent it.  We can
get pretty much the same test coverage without an it's-not-midnight
assumption by moving the does-'now'-work cases into their own test step.

While here, apply commit 47169c25's s/DELETE/TRUNCATE/ change to
timestamptz as well as timestamp (not sure why that didn't
occur to me at the time; the risk of failure is the same).

Back-patch to all supported branches, since the main point is
to get rid of potential buildfarm failures.

Discussion: https://postgr.es/m/14821.1577031117@sss.pgh.pa.us

_GNU_SOURCE is required to get the prototype, so just define that
globally, as was already done in the linux template.

Discussion: https://www.postgresql.org/message-id/flat/6b467edc-4018-521f-ab18-171f098557ca%402ndquadrant.com

This fixes a performance problem introduced by commit 6d7547c2.
ERROR_ACCESS_DENIED is returned in some other cases besides the
delete-pending case considered by that commit; notably, if the
given path names a directory instead of a plain file.  In that
case we'll uselessly loop for 1 second before returning the
failure condition.  That slows down some usage scenarios enough
to cause test timeout failures on our Windows buildfarm critters.

To fix, try to stat() the file, and sleep/loop only if that fails.
It will fail in the delete-pending case, and also in the case where
the deletion completed before we could stat(), so we have the cases
where we want to loop covered.  In the directory case, the stat()
should succeed, letting us exit without a wait.

One case where we'll still wait uselessly is if the access-denied
problem pertains to a directory in the given pathname.  But we don't
expect that to happen in any performance-critical code path.

There might be room to refine this further, but I'll push it now
in hopes of making the buildfarm green again.

Back-patch, like the preceding commit.

Alexander Lakhin and Tom Lane

Discussion: https://postgr.es/m/23073.1576626626@sss.pgh.pa.us

Document why no indentation and why no non-whitespace postfix is
supported.

Backpatch-through: master

Previously it was unclear how COPY FROM handled cases where not all
columns were specified, or if the order didn't match.

Reported-by: pavlo.golub@gmail.com

Discussion: https://postgr.es/m/157487729344.7213.14245726713444755296@wrigleys.postgresql.org

Backpatch-through: 9.4

Per project policy, transient roles created by regression test cases
should be named "regress_something", to reduce the risks of running
such cases against installed servers.  And no such role should ever
be left behind after running a test.

Discussion: https://postgr.es/m/11297.1576868677@sss.pgh.pa.us

We realized years ago that it's better for libpq to accept all
connection parameters syntactically, even if some are ignored or
restricted due to lack of the feature in a particular build.
However, that lesson from the SSL support was for some reason never
applied to the GSSAPI support.  This is causing various buildfarm
members to have problems with a test case added by commit 6136e94d,
and it's just a bad idea from a user-experience standpoint anyway,
so fix it.

While at it, fix some places where parameter-related infrastructure
was added with the aid of a dartboard, or perhaps with the aid of
the anti-pattern "add new stuff at the end".  It should be safe
to rearrange the contents of struct pg_conn even in released
branches, since that's private to libpq (and we'd have to move
some fields in some builds to fix this, anyway).

Back-patch to all supported branches.

Discussion: https://postgr.es/m/11297.1576868677@sss.pgh.pa.us

Most of the MSVC Perl code uses forward slashes for file paths.  Make
the few places that use backslashes the same.  This also helps running
that code on non-Windows.

Previously, the Windows MSVC build generated pg_config.h from a
hard-coded pg_config.h.win32 with some ad hoc postprocessing.  The
pg_config.h.win32 file required manual maintenance and was as a result
frequently out of date.

Instead, have the MSVC build scripts emulate what configure and
config.status do: collect a list of defines and then create
pg_config.h from pg_config.h.in by changing the appropriate lines.

The previous setup was made to support old Windows build systems that
didn't have any text processing capabilities, but the current system
has Perl, so it's not a problem.  pg_config.h.win32 is removed.

In order to try to keep the Windows side of things more up to date in
the future, we now also require that all symbols found in
pg_config.h.in are defined in the MSVC build system.  So if there is a
change in configure that results in a new symbol, an update in
Solution.pm will be required.

The other headers managed by AC_CONFIG_HEADERS in configure, namely
src/include/pg_config_ext.h and
src/interfaces/ecpg/include/ecpg_config.h, get the same treatment, so
this removes even more ad hoc code in the MSVC build scripts.
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://www.postgresql.org/message-id/flat/1441b834-f434-e0bf-46ed-9c4d5c29c2d4%402ndquadrant.com

This was previously not covered by allow_system_table_mods, but now it
is.  The impact in practice is probably low, but this makes it
consistent with most other DDL commands.
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/ee9df1af-c0d8-7c82-5be7-39ce4e3b0a9d%402ndquadrant.com

The PS_USE_NONE case in ps_status.c left a couple of unused variables
exposed.

Discussion: https://www.postgresql.org/message-id/flat/6b467edc-4018-521f-ab18-171f098557ca%402ndquadrant.com

Currently postgres_fdw doesn't permit a non-superuser to connect to a
foreign server without specifying a password, or to use an
authentication mechanism that doesn't use the password. This is to avoid
using the settings and identity of the user running Postgres.

However, this doesn't make sense for all authentication methods. We
therefore allow a superuser to set "password_required 'false'" for user
mappings for the postgres_fdw. The superuser must ensure that the
foreign server won't try to rely solely on the server identity (e.g.
trust, peer, ident) or use an authentication mechanism that relies on the
password settings (e.g. md5, scram-sha-256).

This feature is a prelude to better support for sslcert and sslkey
settings in user mappings.

Author: Craig Ringer.
Discussion: https://postgr.es/m/075135da-545c-f958-fed0-5dcb462d6dae@2ndQuadrant.com

A new function EmitProcSignalBarrier() can be used to emit a global
barrier which all backends that participate in the ProcSignal
mechanism must absorb, and a new function WaitForProcSignalBarrier()
can be used to wait until all relevant backends have in fact
absorbed the barrier.

This can be used to coordinate global state changes, such as turning
checksums on while the system is running.

There's no real client of this mechanism yet, although two are
proposed, but an enum has to have at least one element, so this
includes a placeholder type (PROCSIGNAL_BARRIER_PLACEHOLDER) which
should be replaced by the first real client of this mechanism to
get committed.

Andres Freund and Robert Haas, reviewed by Daniel Gustafsson and,
in earlier versions, by Magnus Hagander.

Discussion: http://postgr.es/m/CA+TgmoZwDk=BguVDVa+qdA6SBKef=PKbaKDQALTC_9qoz1mJqg@mail.gmail.com

The REDO routine for nbtree's xl_btree_vacuum record type hasn't
performed a "pin scan" since commit 3e4b7d87 went in, so clearly there
isn't any point in VACUUM WAL-logging information that won't actually be
used.  Finish off the work of commit 3e4b7d87 (and the closely related
preceding commit 687f2cd7) by removing the code that generates this
unused information.  Also remove the REDO routine code disabled by
commit 3e4b7d87.

Replace the unneeded lastBlockVacuumed field in xl_btree_vacuum with a
new "ndeleted" field.  The new field isn't actually needed right now,
since we could continue to infer the array length from the overall
record length.  However, an upcoming patch to add deduplication to
nbtree needs to add an "items updated" field to xl_btree_vacuum, so we
might as well start being explicit about the number of items now.
(Besides, it doesn't seem like a good idea to leave the xl_btree_vacuum
struct without any fields; the C standard says that that's undefined.)

nbtree VACUUM no longer forces writing a WAL record for the last block
in the index.  Writing out a WAL record with no items for the final
block was supposed to force processing of a lastBlockVacuumed field by a
pin scan.

Bump XLOG_PAGE_MAGIC because xl_btree_vacuum changed.

Discussion: https://postgr.es/m/CAH2-WzmY_mT7UnTzFB5LBQDBkKpdV5UxP3B5bLb7uP%3D%3D6UQJRQ%40mail.gmail.com

Reverts commit 05684c82.

Reported-by: Tom Lane

Discussion: https://postgr.es/m/404.1576770942@sss.pgh.pa.us

Backpatch-through: master

Reported-by: Ranier Vilela

Discussion: https://postgr.es/m/MN2PR18MB2927BB876D12A70FDBE8F35AE3450@MN2PR18MB2927.namprd18.prod.outlook.com

Backpatch-through: master

Discussion: https://postgr.es/m/20191218221326.GA25537@alvherre.pgsql

The "auth-methods" <sect1> used to include descriptions of all our
authentication methods.  Commit 56811e57 promoted its child <sect2>'s
to <sect1>'s, which has advantages but also created some issues:
* The auth-methods page itself is essentially empty/useless.
* Links that pointed to "auth-methods" as a placeholder for all
auth methods were rendered a bit nonsensical.
* DocBook no longer provides a subsection table-of-contents here,
which formerly was a useful if terse summary of available auth methods.

To improve matters, add a handwritten list of all the auth methods.

Per gripe from Dave Cramer.  Back-patch to v11 where the previous
commit came in.

Discussion: https://postgr.es/m/CADK3HH+xQLhcPgg=kWqfogtXGGZr-JdSo=x=WQC0PkAVyxUWyQ@mail.gmail.com

Commit d986d4e8 renamed a variable
but neglected to update the corresponding comment.

Amit Langote

The previous coding imagined that it could call before_shmem_exit()
when a non-exclusive backup began and then remove the previously-added
handler by calling cancel_before_shmem_exit() when that backup
ended. However, this only works provided that nothing else in the
system has registered a before_shmem_exit() hook in the interim,
because cancel_before_shmem_exit() is documented to remove a callback
only if it is the latest callback registered. It also only works
if nothing can ERROR out between the time that sessionBackupState
is reset and the time that cancel_before_shmem_exit(), which doesn't
seem to be strictly true.

To fix, leave the handler installed for the lifetime of the session,
arrange to install it just once, and teach it to quietly do nothing if
there isn't a non-exclusive backup in process.

This is a bug, but for now I'm not going to back-patch, because the
consequences are minor. It's possible to cause a spurious warning
to be generated, but that doesn't really matter. It's also possible
to trigger an assertion failure, but production builds shouldn't
have assertions enabled.

Patch by me, reviewed by Kyotaro Horiguchi, Michael Paquier (who
preferred a different approach, but got outvoted), Fujii Masao,
and Tom Lane, and with comments by various others.

Discussion: http://postgr.es/m/CA+TgmobMjnyBfNhGTKQEDbqXYE3_rXWpc4CM63fhyerNCes3mA@mail.gmail.com

It's basically a variant of Cygwin, so use that template.

Discussion: https://www.postgresql.org/message-id/flat/6b467edc-4018-521f-ab18-171f098557ca%402ndquadrant.com

Commit 7dbfea3c thought it could get
away with removing this, but Thomas Munro reports, on behalf of the
buildfarm, that it's still needed at least on Windows to avoid
compiler warnings.