Skip to content

P
Postgres FD Implementation
Switch branch/tag
  1. 07 Dec, 2021 2 commits
  3. 03 Dec, 2021 2 commits
  5. 02 Dec, 2021 2 commits
    • Tom Lane's avatar
      On Windows, close the client socket explicitly during backend shutdown. · 4cd29285
      Tom Lane authored 
      It turns out that this is necessary to keep Winsock from dropping any
not-yet-sent data, such as an error message explaining the reason for
process termination.  It's pretty weird that the implicit close done
by the kernel acts differently from an explicit close, but it's hard
to argue with experimental results.

Independently submitted by Alexander Lakhin and Lars Kanis (comments
by me, though).  Back-patch to all supported branches.

Discussion: https://postgr.es/m/90b34057-4176-7bb0-0dbb-9822a5f6425b@greiz-reinsdorf.de
Discussion: https://postgr.es/m/16678-253e48d34dc0c376@postgresql.org
      4cd29285
    • Michael Paquier's avatar
      Move into separate file all the SQL queries used in pg_upgrade tests · b6dac98b
      Michael Paquier authored 
      The existing pg_upgrade/test.sh and the buildfarm code have been holding
the same set of SQL queries when doing cross-version upgrade tests to
adapt the objects created by the regression tests before the upgrade
(mostly, incompatible or non-existing objects need to be dropped from
the origin, perhaps re-created).

This moves all those SQL queries into a new, separate, file with a set
of \if clauses to handle the version checks depending on the old version
of the cluster to-be-upgraded.

The long-term plan is to make the buildfarm code re-use this new SQL
file, so as committers are able to fix any compatibility issues in the
tests of pg_upgrade with a refresh of the core code, without having to
poke at the buildfarm client.  Note that this is only able to handle the
main regression test suite, and that nothing is done yet for contrib
modules yet (these have more issues like their database names).

A backpatch down to 10 is done, adapting the version checks as this
script needs to be only backward-compatible, so as it becomes possible
to clean up a maximum amount of code within the buildfarm client.

Author: Justin Pryzby, Michael Paquier
Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com
Backpatch-through: 10
      b6dac98b
  7. 01 Dec, 2021 2 commits
    • Tom Lane's avatar
      Avoid leaking memory during large-scale REASSIGN OWNED BY operations. · 8f4b0200
      Tom Lane authored 
      The various ALTER OWNER routines tend to leak memory in
CurrentMemoryContext.  That's not a problem when they're only called
once per command; but in this usage where we might be touching many
objects, it can amount to a serious memory leak.  Fix that by running
each call in a short-lived context.

(DROP OWNED BY likely has a similar issue, except that you'll probably
run out of lock table space before noticing.  REASSIGN is worth fixing
since for most non-table object types, it won't take any lock.)

Back-patch to all supported branches.  Unfortunately, in the back
branches this helps to only a limited extent, since the sinval message
queue bloats quite a lot in this usage before commit 3aafc030a,
consuming memory more or less comparable to what's actually leaked.
Still, it's clearly a leak with a simple fix, so we might as well fix it.

Justin Pryzby, per report from Guillaume Lelarge

Discussion: https://postgr.es/m/CAECtzeW2DAoioEGBRjR=CzHP6TdL=yosGku8qZxfX9hhtrBB0Q@mail.gmail.com
      8f4b0200
    • Amit Kapila's avatar
      Doc: Add "Attach Partition" limitation during logical replication. · 4b8eec71
      Amit Kapila authored 
      ATTACHing a table into a partition tree whose root is published using a
publication with publish_via_partition_root set to true does not result in
the table's existing contents being replicated. This happens because
subscriber doesn't consider replicating the newly attached partition as
the root table is already in a 'ready' state.

This behavior was introduced in PG13 (83fd4532) where we allowed to
publish partition changes via ancestors.

We can consider fixing this limitation in the future.

Author: Amit Langote
Reviewed-by: Hou Zhijie, Amit Kapila
Backpatch-through: 13
Discussion: https://postgr.es/m/OS0PR01MB5716E97F00732B52DC2BBC2594989@OS0PR01MB5716.jpnprd01.prod.outlook.com
      4b8eec71
  9. 30 Nov, 2021 2 commits
    • Tom Lane's avatar
      Cope with cross-compiling when checking for a random-number source. · 175edafd
      Tom Lane authored 
      Commit 16f96c74 neglected to consider the possibility of cross-compiling,
causing cross-compiles to fail at the configure stage unless you'd
selected --with-openssl.  Since we're now more or less assuming that
/dev/urandom is available everywhere, it seems reasonable to assume
that the cross-compile target has it too, rather than failing.

Per complaint from Vincas Dargis.  Back-patch to v14 where this came in.

Discussion: https://postgr.es/m/0dc14a31-acaf-8cae-0df4-a87339b22bd9@gmail.com
      175edafd
    • Michael Paquier's avatar
      Fix compatibility thinko for fstat() on standard streams in win32stat.c · 5550a9c3
      Michael Paquier authored 
      GetFinalPathNameByHandleA() cannot be used in compilation environments
where _WIN32_WINNT < 0x0600, meaning at least Windows XP used by some
buildfarm members under MinGW that Postgres still needs to support.
This was reported as a compilation warning by the buildfarm, but this is
actually worse than the report as the code would have not worked.

Instead, this switches to GetFileInformationByHandle() that is able to
fail for standard streams and succeed for redirected ones, which is what
we are looking for herein the code emulating fstat().  We also know that
it is able to work in all the environments still supported, thanks to
the existing logic of win32stat.c.

Issue introduced by 10260c7, so backpatch down to 14.

Reported-by: Justin Pryzby, via buildfarm member jacana
Author: Michael Paquier
Reviewed-by: Juan José Santamaría Flecha
Discussion: https://postgr.es/m/20211129050122.GK17618@telsasoft.com
Backpatch-through: 14
      5550a9c3
  11. 29 Nov, 2021 1 commit
  13. 26 Nov, 2021 4 commits
  15. 25 Nov, 2021 3 commits
    • Peter Eisentraut's avatar
      Remove unneeded Python includes · 1cc13b83
      Peter Eisentraut authored 
      Inluding <compile.h> and <eval.h> has not been necessary since Python
2.4, since they are included via <Python.h>.  Morever, <eval.h> is
being removed in Python 3.11.  So remove these includes.
Reviewed-by: default avatarTom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/84884.1637723223%40sss.pgh.pa.us
      1cc13b83
    • Michael Paquier's avatar
      Block ALTER TABLE .. DROP NOT NULL on columns in replica identity index · e415916e
      Michael Paquier authored 
      Replica identities that depend directly on an index rely on a set of
properties, one of them being that all the columns defined in this index
have to be marked as NOT NULL.  There was a hole in the logic with ALTER
TABLE DROP NOT NULL, where it was possible to remove the NOT NULL
property of a column part of an index used as replica identity, so block
it to avoid problems with logical decoding down the road.

The same check was already done columns part of a primary key, so the
fix is straight-forward.

Author: Haiying Tang, Hou Zhijie
Reviewed-by: Dilip Kumar, Michael Paquier
Discussion: https://postgr.es/m/OS0PR01MB6113338C102BEE8B2FFC5BD9FB619@OS0PR01MB6113.jpnprd01.prod.outlook.com
Backpatch-through: 10
      e415916e
    • Michael Paquier's avatar
      Fix fstat() emulation on Windows with standard streams · d2198b45
      Michael Paquier authored 
      The emulation of fstat() in win32stat.c caused two issues with the
existing in-core callers, failing on EINVAL when using a stream as
argument:
- psql's \copy would crash when using a stream.
- pg_recvlogical would fail with -f -.

The tests in copyselect.sql from the main test suite covers the first
case, and there is a TAP test for the second case.  However, in both
cases, as the standard streams are always redirected, automated tests
did not notice those issues, requiring a terminal on Windows to be
reproducible.

This issue has been introduced in bed90759, and the origin of the problem
is that GetFileInformationByHandle() does not work directly on streams,
so this commit adds an extra code path to emulate and return a set of
stats that match best with the reality.  Note that redirected streams
rely on handles that can be queried with GetFileInformationByHandle(),
but we can rely on GetFinalPathNameByHandleA() to detect this case.

Author: Dmitry Koval, Juan José Santamaría Flecha
Discussion: https://postgr.es/m/17288-6b58a91025a8a8a3@postgresql.org
Backpatch-through: 14
      d2198b45
  17. 24 Nov, 2021 6 commits
  19. 23 Nov, 2021 1 commit
    • David Rowley's avatar
      Allow Memoize to operate in binary comparison mode · 6c32c097
      David Rowley authored 
      Memoize would always use the hash equality operator for the cache key
types to determine if the current set of parameters were the same as some
previously cached set.  Certain types such as floating points where -0.0
and +0.0 differ in their binary representation but are classed as equal by
the hash equality operator may cause problems as unless the join uses the
same operator it's possible that whichever join operator is being used
would be able to distinguish the two values.  In which case we may
accidentally return in the incorrect rows out of the cache.

To fix this here we add a binary mode to Memoize to allow it to the
current set of parameters to previously cached values by comparing
bit-by-bit rather than logically using the hash equality operator.  This
binary mode is always used for LATERAL joins and it's used for normal
joins when any of the join operators are not hashable.

Reported-by: Tom Lane
Author: David Rowley
Discussion: https://postgr.es/m/3004308.1632952496@sss.pgh.pa.us
Backpatch-through: 14, where Memoize was added
      6c32c097
  21. 22 Nov, 2021 5 commits
    • Tom Lane's avatar
      Adjust pg_dump's priority ordering for casts. · 0fdf6747
      Tom Lane authored 
      When a stored expression depends on a user-defined cast, the backend
records the dependency as being on the cast's implementation function
--- or indeed, if there's no cast function involved but just
RelabelType or CoerceViaIO, no dependency is recorded at all.  This
is problematic for pg_dump, which is at risk of dumping things in the
wrong order leading to restore failures.  Given the lack of previous
reports, the risk isn't that high, but it can be demonstrated if the
cast is used in some view whose rowtype is then used as an input or
result type for some other function.  (That results in the view
getting hoisted into the functions portion of the dump, ahead of
the cast.)

A logically bulletproof fix for this would require including the
cast's OID in the parsed form of the expression, whence it could be
extracted by dependency.c, and then the stored dependency would force
pg_dump to do the right thing.  Such a change would be fairly invasive,
and certainly not back-patchable.  Moreover, since we'd prefer that
an expression using cast syntax be equal() to one doing the same
thing by explicit function call, the cast OID field would have to
have special ignored-by-comparisons semantics, making things messy.

So, let's instead fix this by a very simple hack in pg_dump: change
the object-type priority order so that casts are initially sorted
before functions, immediately after types.  This fixes the problem
in a fairly direct way for casts that have no implementation function.
For those that do, the implementation function will be hoisted to just
before the cast by the dependency sorting step, so that we still have
a valid dump order.  (I'm not sure that this provides a full guarantee
of no problems; but since it's been like this for many years without
any previous reports, this is probably enough to fix it in practice.)

Per report from Дмитрий Иванов.
Back-patch to all supported branches.

Discussion: https://postgr.es/m/CAPL5KHoGa3uvyKp6z6m48LwCnTsK+LRQ_mcA4uKGfqAVSEjV_A@mail.gmail.com
      0fdf6747
    • Tom Lane's avatar
      Pacify perlcritic. · 72842a57
      Tom Lane authored 
      Per buildfarm.
      72842a57
    • Tom Lane's avatar
      Fix pg_dump --inserts mode for generated columns with dropped columns. · aedc4600
      Tom Lane authored 
      If a table contains a generated column that's preceded by a dropped
column, dumpTableData_insert failed to account for the dropped
column, and would emit DEFAULT placeholder(s) in the wrong column(s).
This resulted in failures at restore time.  The default COPY code path
did not have this bug, likely explaining why it wasn't noticed sooner.

While we're fixing this, we can be a little smarter about the
situation: (1) avoid unnecessarily fetching the values of generated
columns, (2) omit generated columns from the output, too, if we're
using --column-inserts.  While these modes aren't expected to be
as high-performance as the COPY path, we might as well be as
efficient as we can; it doesn't add much complexity.

Per report from Дмитрий Иванов.
Back-patch to v12 where generated columns came in.

Discussion: https://postgr.es/m/CAPL5KHrkBniyQt5e1rafm5DdXvbgiiqfEQEJ9GjtVzN71Jj5pA@mail.gmail.com
      aedc4600
    • Tom Lane's avatar
      Probe $PROVE not $PERL while checking for modules needed by TAP tests. · e9af18c6
      Tom Lane authored 
      Normally "prove" and "perl" come from the same Perl installation,
but we support the case where they don't (mainly because the MSys
buildfarm animals need this).  In that case, AX_PROG_PERL_MODULES
is completely the wrong thing to use, because it's checking what
"perl" has.  Instead, make a little TAP test script including the
required modules, and run that under "prove".

We don't need ax_prog_perl_modules.m4 at all after this change,
so remove it.

Back-patch to all supported branches, for the buildfarm's benefit.
(In v10, this also back-patches the effects of commit 264eb03a.)

Andrew Dunstan and Tom Lane, per an observation by Noah Misch

Discussion: https://postgr.es/m/E1moZHS-0002Cu-Ei@gemulon.postgresql.org
      e9af18c6
    • Alvaro Herrera's avatar
      Add missing words in comment · c985a43d
      Alvaro Herrera authored 
      Reported by Zhihong Yu.

Discussion: https://postgr.es/m/CALNJ-vR6uZivg_XkB1zKjEXeyZDEgoYanFXB-++1kBT9yZQoUw@mail.gmail.com
      c985a43d
  23. 21 Nov, 2021 1 commit
    • Tom Lane's avatar
      pg_receivewal, pg_recvlogical: allow canceling initial password prompt. · 3bd7556b
      Tom Lane authored 
      Previously it was impossible to terminate these programs via control-C
while they were prompting for a password.  We can fix that trivially
for their initial password prompts, by moving setup of the SIGINT
handler from just before to just after their initial GetConnection()
calls.

This fix doesn't permit escaping out of later re-prompts, but those
should be exceedingly rare, since the user's password or the server's
authentication setup would have to have changed meanwhile.  We
considered applying a fix similar to commit 46d665bc2, but that
seemed more complicated than it'd be worth.  Moreover, this way is
back-patchable, which that wasn't.

The misbehavior exists in all supported versions, so back-patch to all.

Tom Lane and Nathan Bossart

Discussion: https://postgr.es/m/747443.1635536754@sss.pgh.pa.us
      3bd7556b
  25. 20 Nov, 2021 1 commit
    • Tom Lane's avatar
      Fix SP-GiST scan initialization logic for binary-compatible cases. · 6d07cbc5
      Tom Lane authored 
      Commit ac9099fc rearranged the logic in spgGetCache() that determines
the index's attType (nominal input data type) and leafType (actual
type stored in leaf index tuples).  Turns out this broke things for
the case where (a) the actual input data type is different from the
nominal type, (b) the opclass's config function leaves leafType
defaulted, and (c) the opclass has no "compress" function.  (b) caused
us to assign the actual input data type as leafType, and then since
that's not attType, we complained that a "compress" function is
required.  For non-polymorphic opclasses, condition (a) arises in
binary-compatible cases, such as using SP-GiST text_ops for a varchar
column, or using any opclass on a domain over its nominal input type.

To fix, use attType for leafType when the index's declared column type
is different from but binary-compatible with attType.  Do this only in
the defaulted-leafType case, to avoid overriding any explicit
selection made by the opclass.

Per bug #17294 from Ilya Anfimov.  Back-patch to v14.

Discussion: https://postgr.es/m/17294-8f6c7962ce877edc@postgresql.org
      6d07cbc5
  27. 19 Nov, 2021 1 commit
    • Amit Kapila's avatar
      Fix parallel operations that prevent oldest xmin from advancing. · ead49ebc
      Amit Kapila authored 
      While determining xid horizons, we skip over backends that are running
Vacuum. We also ignore Create Index Concurrently, or Reindex Concurrently
for the purposes of computing Xmin for Vacuum. But we were not setting the
flags corresponding to these operations when they are performed in
parallel which was preventing Xid horizon from advancing.

The optimization related to skipping Create Index Concurrently, or Reindex
Concurrently operations was implemented in PG-14 but the fix is the same
for the Parallel Vacuum as well so back-patched till PG-13.

Author: Masahiko Sawada
Reviewed-by: Amit Kapila
Backpatch-through: 13
Discussion: https://postgr.es/m/CAD21AoCLQqgM1sXh9BrDFq0uzd3RBFKi=Vfo6cjjKODm0Onr5w@mail.gmail.com
      ead49ebc
  29. 18 Nov, 2021 3 commits
    • Tom Lane's avatar
      Use appropriate -Wno-warning switches when compiling bitcode. · ed1c261a
      Tom Lane authored 
      We use "clang" to compile bitcode files for LLVM inlining.  That might
be different from the build's main C compiler, so it needs its own set
of compiler flags.  To simplify configure, we don't bother adding any
-W switches to that flag set; there's little need since the main build
will show us any warnings.  However, if we don't want to see unwanted
warnings, we still have to add any -Wno-warning switches we'd normally
use with clang.

This escaped notice before commit 9ff47ea41, which tried to add
-Wno-compound-token-split-by-macro; buildfarm animals using mismatched
CC and CLANG still showed those warnings.  I'm not sure why we never
saw any effects from the lack of -Wno-unused-command-line-argument
(maybe that's only activated by -Wall?).  clang does not currently
support -Wno-format-truncation or -Wno-stringop-truncation, although
in the interests of future-proofing and consistency I included tests
for those.

Back-patch to v11 where we started building bitcode files.

Discussion: https://postgr.es/m/2921539.1637254619@sss.pgh.pa.us
      ed1c261a
    • Michael Paquier's avatar
      Fix quoting of ACL item in table for upgrade binary compatibility checks · 048f3ee6
      Michael Paquier authored 
      Per buildfarm member prion, that runs the regression tests under a role
name that uses a hyphen.  Issue introduced by 835bcba.

Discussion: https://postgr.es/m/YZW4MvzCZ+hQ34vw@paquier.xyz
Backpatch-through: 12
      048f3ee6
    • Michael Paquier's avatar
      Add table to regression tests for binary-compatibility checks in pg_upgrade · cf3d79aa
      Michael Paquier authored 
      This commit adds to the main regression test suite a table with all
the in-core data types (some exceptions apply).  This table is not
dropped, so as pg_upgrade would be able to check the binary
compatibility of the types tracked in the table.  If a new type is added
in core, this part of the tests would need a refresh but the tests are
designed to fail if that were to happen.

As this is useful for upgrades and that these rely on the objects
created in the regression test suite of the old version upgraded from,
a backpatch down to 12 is done, which is the last point where a binary
incompatible change has been done (7c15cef8).  This will hopefully be
enough to find out if something gets broken during the development of a
new version of Postgres, so as it is possible to take actions in
pg_upgrade itself in this case (like 0ccfc282 for sql_identifier).

An area that is not covered yet is related to external modules, which
may create their own types.  The testing infrastructure of pg_upgrade is
not integrated yet with the external modules stored in core
(src/test/modules/ or contrib/, all use the same database name for their
tests so there would be an overlap).  This could be improved in the
future.

Author: Justin Pryzby
Reviewed-by: Jacob Champion, Peter Eisentraut, Tom Lane, Michael Paquier
Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com
Backpatch-through: 12
      cf3d79aa
  31. 17 Nov, 2021 4 commits
    • Tom Lane's avatar
      Clean up error handling in pg_basebackup's walmethods.c. · 53c4a580
      Tom Lane authored 
      The error handling here was a mess, as a result of a fundamentally
bad design (relying on errno to keep its value much longer than is
safe to assume) as well as a lot of just plain sloppiness, both as
to noticing errors at all and as to reporting the correct errno.
Moreover, the recent addition of LZ4 compression broke things
completely, because liblz4 doesn't use errno to report errors.

To improve matters, keep the error state in the DirectoryMethodData or
TarMethodData struct, and add a string field so we can handle cases
that don't set errno.  (The tar methods already had a version of this,
but it can be done more efficiently since all these cases use a
constant error string.)  Make the dir and tar methods handle errors
in basically identical ways, which they didn't before.

This requires copying errno into the state struct in a lot of places,
which is a bit tedious, but it has the virtue that we can get rid of
ad-hoc code to save and restore errno in a number of places ... not
to mention that it fixes other places that should've saved/restored
errno but neglected to.

In passing, fix some pointlessly static buffers to be ordinary
local variables.

There remains an issue about exactly how to handle errors from
fsync(), but that seems like material for its own patch.

While the LZ4 problems are new, all the rest of this is fixes for
old bugs, so backpatch to v10 where walmethods.c was introduced.

Patch by me; thanks to Michael Paquier for review.

Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us
      53c4a580
    • Tom Lane's avatar
      Handle close() failures more robustly in pg_dump and pg_basebackup. · 6b413b41
      Tom Lane authored 
      Coverity complained that applying get_gz_error after a failed gzclose,
as we did in one place in pg_basebackup, is unsafe.  I think it's
right: it's entirely likely that the call is touching freed memory.
Change that to inspect errno, as we do for other gzclose calls.

Also, be careful to initialize errno to zero immediately before any
gzclose() call where we care about the error status.  (There are
some calls where we don't, because we already failed at some previous
step.)  This ensures that we don't get a misleadingly irrelevant
error code if gzclose() fails in a way that doesn't set errno.
We could work harder at that, but it looks to me like all such cases
are basically can't-happen if we're not misusing zlib, so it's
not worth the extra notational cruft that would be required.

Also, fix several places that simply failed to check for close-time
errors at all, mostly at some remove from the close or gzclose itself;
and one place that did check but didn't bother to report the errno.

Back-patch to v12.  These mistakes are older than that, but between
the frontend logging API changes that happened in v12 and the fact
that frontend code can't rely on %m before that, the patch would need
substantial revision to work in older branches.  It doesn't quite
seem worth the trouble given the lack of related field complaints.

Patch by me; thanks to Michael Paquier for review.

Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us
      6b413b41
    • Tom Lane's avatar
      Fix display of SQL-standard function's arguments in INSERT/SELECT. · 5d5779ae
      Tom Lane authored 
      If a SQL-standard function body contains an INSERT ... SELECT statement,
any function parameters referenced within the SELECT were always printed
in $N style, rather than using the parameter name if any.  While not
strictly incorrect, this wasn't the intention, and it's inconsistent
with the way that such parameters would be printed in any other kind
of statement.

The cause is that the recursion to get_query_def from
get_insert_query_def neglected to pass down the context->namespaces
list, passing constant NIL instead.  This is a very ancient oversight,
but AFAICT it had no visible consequences before commit e717a9a1
added an outermost namespace with function parameters.  We don't allow
INSERT ... SELECT as a sub-query, except in a top-level WITH clause,
where it couldn't contain any outer references that might need to access
upper namespaces.  So although that's arguably a bug, I don't see any
point in changing it before v14.

In passing, harden the code added to get_parameter by e717a9a1 so that
it won't crash if a PARAM_EXTERN Param appears in an unexpected place.

Per report from Erki Eessaar.  Code fix by me, regression test case
by Masahiko Sawada.

Discussion: https://postgr.es/m/AM9PR01MB8268347BED344848555167FAFE949@AM9PR01MB8268.eurprd01.prod.exchangelabs.com
      5d5779ae
    • Daniel Gustafsson's avatar
      Doc: add see-also references to CREATE PUBLICATION. · 51d42136
      Daniel Gustafsson authored 
      The "See also" section on the reference page for CREATE PUBLICATION
didn't match the cross references on CREATE SUBSCRIPTION and their
ALTER counterparts. Fixed by adding an xref to the CREATE and ALTER
SUBSCRIPTION pages.  Backpatch down to v10 where CREATE PUBLICATION
was introduced.

Author: Peter Smith <smithpb2250@gmail.com>
Reviewed-by: default avatarMasahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://postgr.es/m/CAHut+PvGWd3-Ktn96c-z6uq-8TGVVP=TPOkEovkEfntoo2mRhw@mail.gmail.com
Backpatch-through: 10
      51d42136