The core idea of this patch is to make the parser generate join alias
Vars (that is, ones with varno pointing to a JOIN RTE) only when the
alias Var is actually different from any raw join input, that is a type
coercion and/or COALESCE is necessary to generate the join output value.
Otherwise just generate varno/varattno pointing to the relevant join
input column.

In effect, this means that the planner's flatten_join_alias_vars()
transformation is already done in the parser, for all cases except
(a) columns that are merged by JOIN USING and are transformed in the
process, and (b) whole-row join Vars.  In principle that would allow
us to skip doing flatten_join_alias_vars() in many more queries than
we do now, but we don't have quite enough infrastructure to know that
we can do so --- in particular there's no cheap way to know whether
there are any whole-row join Vars.  I'm not sure if it's worth the
trouble to add a Query-level flag for that, and in any case it seems
like fit material for a separate patch.  But even without skipping the
work entirely, this should make flatten_join_alias_vars() faster,
particularly where there are nested joins that it previously had to
flatten recursively.

An essential part of this change is to replace Var nodes'
varnoold/varoattno fields with varnosyn/varattnosyn, which have
considerably more tightly-defined meanings than the old fields: when
they differ from varno/varattno, they identify the Var's position in
an aliased JOIN RTE, and the join alias is what ruleutils.c should
print for the Var.  This is necessary because the varno change
destroyed ruleutils.c's ability to find the JOIN RTE from the Var's
varno.

Another way in which this change broke ruleutils.c is that it's no
longer feasible to determine, from a JOIN RTE's joinaliasvars list,
which join columns correspond to which columns of the join's immediate
input relations.  (If those are sub-joins, the joinaliasvars entries
may point to columns of their base relations, not the sub-joins.)
But that was a horrid mess requiring a lot of fragile assumptions
already, so let's just bite the bullet and add some more JOIN RTE
fields to make it more straightforward to figure that out.  I added
two integer-List fields containing the relevant column numbers from
the left and right input rels, plus a count of how many merged columns
there are.

This patch depends on the ParseNamespaceColumn infrastructure that
I added in commit 5815696b.  The biggest bit of code change is
restructuring transformFromClauseItem's handling of JOINs so that
the ParseNamespaceColumn data is propagated upward correctly.

Other than that and the ruleutils fixes, everything pretty much
just works, though some processing is now inessential.  I grabbed
two pieces of low-hanging fruit in that line:

1. In find_expr_references, we don't need to recurse into join alias
Vars anymore.  There aren't any except for references to merged USING
columns, which are more properly handled when we scan the join's RTE.
This change actually fixes an edge-case issue: we will now record a
dependency on any type-coercion function present in a USING column's
joinaliasvar, even if that join column has no references in the query
text.  The odds of the missing dependency causing a problem seem quite
small: you'd have to posit somebody dropping an implicit cast between
two data types, without removing the types themselves, and then having
a stored rule containing a whole-row Var for a join whose USING merge
depends on that cast.  So I don't feel a great need to change this in
the back branches.  But in theory this way is more correct.

2. markRTEForSelectPriv and markTargetListOrigin don't need to recurse
into join alias Vars either, because the cases they care about don't
apply to alias Vars for USING columns that are semantically distinct
from the underlying columns.  This removes the only case in which
markVarForSelectPriv could be called with NULL for the RTE, so adjust
the comments to describe that hack as being strictly internal to
markRTEForSelectPriv.

catversion bump required due to changes in stored rules.

Discussion: https://postgr.es/m/7115.1577986646@sss.pgh.pa.us

This tells you about allocations that have been made from the main
shared memory segment. The original patch also tried to show information
about dynamic shared memory allocation as well, but I decided to
leave that problem for another time.

Andres Freund and Robert Haas, reviewed by Michael Paquier, Marti
Raudsepp, Tom Lane, Álvaro Herrera, and Kyotaro Horiguchi.

Discussion: http://postgr.es/m/20140504114417.GM12715@awork2.anarazel.de

Per the buildfarm, via Michael Paquier.

Discussion: http://postgr.es/m/20200108032648.GE3413@paquier.xyz

Reported-by: Octopus ZHANG
Author: Daniel Gustafsson

We currently have several sets of files generated from data provided
by Unicode.  These all have ad hoc rules and instructions for updating
when new Unicode versions appear, and it's not done consistently.

This patch centralizes and automates the process and makes it part of
the release checklist.  The Unicode and CLDR versions are specified in
Makefile.global.in.  There is a new make target "update-unicode" that
downloads all the relevant files and runs the generation script.

There is also a new script for generating the table of combining
characters for ucs_wcwidth().  That table is now in a separate include
file rather than hardcoded into the middle of other code.  This is
based on the script that was used for generating
d8594d12, but the script itself wasn't
committed at that time.
Reviewed-by: John Naylor <john.naylor@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/c8d05f42-443e-6c23-819b-05b31759a37c@2ndquadrant.com

This allows different users to authenticate with different certificates.

Author: Craig Ringer

Change the exception syntax used in the tests to use the more current

    except Exception as ex:

rather than the old

    except Exception, ex:

Since support for Python <2.6 has been removed, all supported versions
now support the new style, and we can save one step in the Python 3
compatibility conversion.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/98b69261-298c-13d2-f34d-836fd9c29b21%402ndquadrant.com

Supporting very old Python versions is a maintenance burden,
especially with the several variant test files to maintain for Python
<2.6.

Since we have dropped support for older OpenSSL versions in
7b283d0e, RHEL 5 is now effectively
desupported, and that was also the only mainstream operating system
still using Python versions before 2.6, so it's a good time to drop
those as well.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/98b69261-298c-13d2-f34d-836fd9c29b21%402ndquadrant.com

Make the value null only at pg_stat_activity-output time, as suggested
by Tom Lane, instead of messing with the internal state.  This should
appease buildfarm members with force_parallel_mode=regress, which are
running parallel queries on logical replication walsenders.

The fact that walsenders can run parallel queries should perhaps be
studied more carefully, but for the moment let's get rid of the red
blots in buildfarm.

Backpatch to pg10, like the previous commit.

Discussion: https://postgr.es/m/30804.1578438763@sss.pgh.pa.us

Use the parser's standard type coercion machinery to convert the
output column(s) of a SQL function's final SELECT or RETURNING
to the type(s) they should have according to the function's declared
result type.  We'll allow any case where an assignment-level
coercion is available.  Previously, we failed unless the required
coercion was a binary-compatible one (and the documentation ignored
this, falsely claiming that the types must match exactly).

Notably, the coercion now accounts for typmods, so that cases where
a SQL function is declared to return a composite type whose columns
are typmod-constrained now behave as one would expect.  Arguably
this aspect is a bug fix, but the overall behavioral change here
seems too large to consider back-patching.

A nice side-effect is that functions can now be inlined in a
few cases where we previously failed to do so because of type
mismatches.

Discussion: https://postgr.es/m/18929.1574895430@sss.pgh.pa.us

The original comment was a bit confusing, pointed out by Alvaro Herrera.

Thread: https://postgr.es/m/20191224151520.GA16435%40alvherre.pgsql

ALTER TABLE failed if a column referenced in a GENERATED expression
had been added or changed in type earlier in the ALTER command.
That's because the GENERATED expression needs to be evaluated
against the table's updated tuples, but it was being evaluated
against the original tuples.  (Fortunately the executor has adequate
cross-checks to notice the mismatch, so we just got an obscure error
message and not anything more dangerous.)

Per report from Andreas Joseph Krogh.  Back-patch to v12 where
GENERATED was added.

Discussion: https://postgr.es/m/VisenaEmail.200.231b0a41523275d0.16ea7f800c7@tc7-visena

Author: Fabien COELHO <coelho@cri.ensmp.fr>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.1912241100390.3339@pseudo

This reverts commit a052f6cb, following complains from Robert Haas and
Tom Lane.  Backpatch down to 9.4, like the previous commit.

Discussion: https://postgr.es/m/CA+TgmobL4npEX5=E5h=5Jm_9mZun3MT39Kq2suJFVeamc9skSQ@mail.gmail.com
Backpatch-through: 9.4

Failures in allocations could lead to crashes with NULL pointer
dereferences .  Memory context TopMemoryContext is used instead to keep
alive the plans allocated in the session.  A more specific context could
be used here, but this is left for later.

Reported-by: Jian Zhang
Author: Michael Paquier
Reviewed-by: Tom Lane, Andres Freund
Discussion: https://postgr.es/m/16190-70181c803641c3dc@postgresql.org

Returning a non-NULL time is pointless, sinc a walsender is not a
process that would be running normal transactions anyway, but the code
was unintentionally exposing the process start time intermittently,
which was not only bogus but it also confused monitoring systems looking
for idle transactions.  Fix by avoiding all updates in walsenders.

Backpatch to 11, where walsenders started appearing in pg_stat_activity.

Reported-by: Tomas Vondra
Discussion: https://postgr.es/m/20191209234409.exe7osmyalwkt5j4@development

Instead of always calling heap_fetch_toast_slice during detoasting,
invoke a table AM callback which, when the toast table is a heap
table, will be heap_fetch_toast_slice.

This makes it possible for a table AM other than heap to be used
as a TOAST table. It also completes the series of commits intended
to improve the interaction of tableam with TOAST that began with
commit 8b94dab0; detoast.c is
now, hopefully, fully AM-independent.

Patch by me, reviewed by Andres Freund and Peter Eisentraut.

Discussion: http://postgr.es/m/CA+TgmoZv-=2iWM4jcw5ZhJeL18HF96+W1yJeYrnGMYdkFFnEpQ@mail.gmail.com

Previously, the toast table had to be implemented by the same AM that
was used for the main table, which was bad, because the detoasting
code won't work with anything but heap. This commit doesn't fix the
latter problem, although there's another patch coming which does,
but it does let you pick something that works (i.e. heap, right now).

Patch by me, reviewed by Andres Freund.

Discussion: http://postgr.es/m/CA+TgmoZv-=2iWM4jcw5ZhJeL18HF96+W1yJeYrnGMYdkFFnEpQ@mail.gmail.com

This one-line change provoked a lot of discussion, but ultimately
the consensus seems to be that allowing a larger value might be
useful to somebody, and probably won't hurt anyone who chooses
not to take advantage of the higher maximum limit.

Vyacheslav Makarov, reviewed by many people.

Discussion: http://postgr.es/m/7b5ecc5a9991045e2f13c84e3047541d@postgrespro.ru

Instead of hard-wiring the netmask as /32, allow it to be specified
where we specify the server address.  This will ease changing the
test to use IPv6, when/if somebody wants to do that.

Also remove the hard-wired pg_hba.conf entries for IPv6 (::1/128).
These have never had any usefulness, because the client side
of the tests has always explicitly connected to $SERVERHOSTADDR
which has always been set to IPv4 (127.0.0.1).  All they accomplish
is to break the test on non-IPv6-supporting hosts, and besides
that they violate the express intent of the code to minimize the
server's range of allowed connections.

This could be back-patched, perhaps, but for now I don't see
a need to.

Discussion: https://postgr.es/m/1899.1578356089@sss.pgh.pa.us

Since the WAL flush position only moves forward, it's safe to cache
its previous value within each walsender process, and update from
shared memory only once we've caught up to the previously-seen value.
When there are many active walsenders, this makes for a very significant
reduction in the amount of contention on the XLogCtl->info_lck spinlock.

This patch also adjusts the logic so that we update our idea of the
flush position after processing a WAL record, rather than beforehand.
This may cause us to realize we're not caught up when the preceding
coding would've thought that we were, but that seems all to the good;
it may avoid a useless sleep-and-wakeup cycle.

Back-patch to v12.  The contention problem exists in prior branches,
but it's much less severe (due to inefficiencies elsewhere) so there
seems no need to take any risk of back-patching further.

Pierre Ducroquet, reviewed by Julien Rouhaud

Discussion: https://postgr.es/m/2931018.Vxl9zapr77@pierred-pdoc

These allow better control of trailing zeroes in numeric values.

Pavel Stehule, based on an old proposal of Marko Tiikkaja's;
review by Karl Pinc

Discussion: https://postgr.es/m/CAFj8pRDjs-navGASeF0Wk74N36YGFJ+v=Ok9_knRa7vDc-qugg@mail.gmail.com

The logical replication apply worker did not fire per-column update
triggers because the updatedCols bitmap in the RTE was not populated.
This fixes that.
Reviewed-by: Euler Taveira <euler@timbira.com.br>
Discussion: https://www.postgresql.org/message-id/flat/21673e2d-597c-6afe-637e-e8b10425b240%402ndquadrant.com

Support is out of scope from all the major vendors for these versions
(for example RHEL5 uses a version based on 0.9.8, and RHEL6 uses 1.0.1),
and it created some extra maintenance work.  Upstream has stopped
support of 0.9.8 in December 2015 and of 1.0.0 in February 2016.

Since b1abfec8, note that the default SSL protocol version set with
ssl_min_protocol_version is TLSv1.2, whose support was added in OpenSSL
1.0.1, so there is no point to enforce ssl_min_protocol_version to TLSv1
in the SSL tests.

Author: Michael Paquier
Reviewed-by: Daniel Gustafsson, Tom Lane
Discussion: https://postgr.es/m/20191205083252.GE5064@paquier.xyz

The fastpath insert optimization's incomplete split flag Assert() is
redundant.  We'll reach the more general Assert() within
_bt_findinsertloc() in all cases. (Besides, Assert()'ing that the
rightmost page doesn't have the flag set never made much sense.)

Use qr// syntax for regex values.
Include the regex that failed to match in diagnostic reports.

Dagfinn Ilmari Mannsåker

Discussion: https://postgr.es/m/87k16610xk.fsf@wibble.ilmari.org

Per suggestion from Tom Lane.
Discussion: https://postgr.es/m/flat/20191230.093451.1762483750956466101.t-ishii%40sraoss.co.jp

The true explanation for Peter Geoghegan's trouble report turns out
to be that he has a ~/.inputrc that affects readline's behavior
enough to break this test.  Prevent readline from reading that file.

Also, the best way to prevent TERM from affecting the results seems
to be to unset it altogether, not to set it to "xterm".  The latter
choice licenses readline to emit xterm escape sequences, and there's
a lot of variation in exactly what it will emit.

Revert changes that attempted to account exactly for xterm escape
sequences.  We shouldn't need that with TERM unset, and it was not
looking like a maintainable solution anyway.

Discussion: https://postgr.es/m/23181.1578167938@sss.pgh.pa.us

Right at the moment, this is making things worse not better in the
buildfarm.  I'm not happy with anything about the current state,
but let's at least try to have a green buildfarm report while further
investigation continues.

Discussion: https://postgr.es/m/23181.1578167938@sss.pgh.pa.us

I'm curious to see what values are prevailing in the buildfarm.

Discussion: https://postgr.es/m/23181.1578167938@sss.pgh.pa.us

It has undefined behavior.  Follow the precedent of commit
9a9473f3.  No back-patch, since the
master branch alone has this function.

Discussion: https://postgr.es/m/20191229070221.GA13873@gust.leadboat.com

Depending on as-yet-incompletely-explained factors, readline/libedit
might choose to emit screen-control escape sequences as part of
repainting the display.  I'd tried to make the test patterns avoid
matching parts of the output that are likely to contain such, but
it seems that there's really no way around matching them explicitly
in some places, unless we want to just give up testing some behaviors
such as display of alternatives.

Per report from Peter Geoghegan.

Discussion: https://postgr.es/m/CAH2-WznPzfWHu8PQwv1Qjpf4wQVPaaWpoO5NunFz9zsYKB4uJA@mail.gmail.com

Pass ParseState into the functions called from
standard_ProcessUtility() instead passing the query string and query
environment separately.  No functionality change, but it makes the
notation consistent.  We had already started moving things into
that direction piece by piece, and this completes it.
Reviewed-by: Pavel Stehule <pavel.stehule@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/6e7aa4a1-be6a-1a75-b1f9-83a678e5184a@2ndquadrant.com

Commit 558a9165 taught _bt_delitems_delete() to produce its own XID
horizon on the primary.  Standbys no longer needed to generate their own
latestRemovedXid, since they could just use the explicitly logged value
from the primary instead.  The deleted offset numbers array from the
xl_btree_delete WAL record was no longer used by the REDO routine for
anything other than deleting the items.

This enables a minor optimization:  We now treat the array as buffer
state, not generic WAL data, following _bt_delitems_vacuum()'s example.
This should be a minor win, since it allows us to avoid including the
deleted items array in cases where XLogInsert() stores the whole buffer
anyway.  The primary goal here is to make the code more maintainable,
though.  Removing inessential differences between the two functions
highlights the fundamental differences that remain.

Also change xl_btree_delete to use uint32 for the size of the array of
item offsets being deleted.  This brings xl_btree_delete closer to
xl_btree_vacuum.  Furthermore, it seems like a good idea to use an
explicit-width integer type (the field was previously an "int").

Bump XLOG_PAGE_MAGIC because xl_btree_delete changed.

Discussion: https://postgr.es/m/CAH2-Wzkz4TjmezzfAbaV1zYrh=fr0bCpzuJTvBe5iUQ3aUPsCQ@mail.gmail.com

Escape non-printable characters in failure reports, by using Data::Dumper
in Useqq mode.  Also, bump $Test::Builder::Level so the diagnostic
references the calling line, and use diag() instad of note(),
so it shows even in non-verbose mode (per request from Christoph Berg).

Also, give up on trying to test for the specific way that readline
chooses to overwrite existing text in the \DRD -> \drds test.
There are too many variants, it seems, at least on the libedit
side of things.

Dagfinn Ilmari Mannsåker and Tom Lane

Discussion: https://postgr.es/m/20200103110128.GA28967@msg.df7cb.de

Debian unstable is shipping a broken version of libedit: it de-escapes
words before passing them to the application's tab completion function,
preventing us from recognizing backslash commands.  Fortunately,
we have enough information available to dig the original text out of
rl_line_buffer, so ignore the string argument and do that.

I view this as a temporary workaround to get the affected buildfarm
members back to green in the wake of 7c015045.  I hope we can get
rid of it once somebody fixes Debian's libedit; hence, no back-patch,
at least for now.

Discussion: https://postgr.es/m/20200103110128.GA28967@msg.df7cb.de

Author: Fabien COELHO <coelho@cri.ensmp.fr>

Reported-by: Jon Jensen
Author: Jon Jensen
Reviewed-by: Amit Kapila and Robert Haas
Backpatch-through: 10
Discussion: https://postgr.es/m/nycvar.YSQ.7.76.1912301807510.9899@ybpnyubfg

Adjust a comment that describes how alignment of the new left page high
key works in btree_xlog_split(), the nbtree page split REDO routine.
The wording used before commit 2c03216d is much clearer, so go back
to that.

Satisfy perlcritic, mostly.  Per buildfarm.