1. 28 Jul, 2022 2 commits
    • Alvaro Herrera's avatar
      Fix replay of create database records on standby · a3aacb7c
      Alvaro Herrera authored
      Crash recovery on standby may encounter missing directories
      when replaying database-creation WAL records.  Prior to this
      patch, the standby would fail to recover in such a case;
      however, the directories could be legitimately missing.
      Consider the following sequence of commands:
      
          CREATE DATABASE
          DROP DATABASE
          DROP TABLESPACE
      
      If, after replaying the last WAL record and removing the
      tablespace directory, the standby crashes and has to replay the
      create database record again, crash recovery must be able to continue.
      
      A fix for this problem was already attempted in 49d9cfc68bf4, but it
      was reverted because of design issues.  This new version is based
      on Robert Haas' proposal: any missing tablespaces are created
      during recovery before reaching consistency.  Tablespaces
      are created as real directories, and should be deleted
      by later replay.  CheckRecoveryConsistency ensures
      they have disappeared.
      
      The problems detected by this new code are reported as PANIC,
      except when allow_in_place_tablespaces is set to ON, in which
      case they are WARNING.  Apart from making tests possible, this
      gives users an escape hatch in case things don't go as planned.
      
      Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
      Author: Asim R Praveen <apraveen@pivotal.io>
      Author: Paul Guo <paulguo@gmail.com>
      Reviewed-by: Anastasia Lubennikova <lubennikovaav@gmail.com> (older versions)
      Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com> (older versions)
      Reviewed-by: default avatarMichaël Paquier <michael@paquier.xyz>
      Diagnosed-by: default avatarPaul Guo <paulguo@gmail.com>
      Discussion: https://postgr.es/m/CAEET0ZGx9AvioViLf7nbR_8tH9-=27DN5xWJ2P9-ROH16e4JUA@mail.gmail.com
      a3aacb7c
    • Thomas Munro's avatar
      Fix get_dirent_type() for symlinks on MinGW/MSYS. · 5ad478c9
      Thomas Munro authored
      On Windows with MSVC, get_dirent_type() was recently made to return
      DT_LNK for junction points by commit 9d3444dc, which fixed some
      defective dirent.c code.
      
      On Windows with Cygwin, get_dirent_type() already worked for symlinks,
      as it does on POSIX systems, because Cygwin has its own fake symlinks
      that behave like POSIX (on closer inspection, Cygwin's dirent has the
      BSD d_type extension but it's probably always DT_UNKNOWN, so we fall
      back to lstat(), which understands Cygwin symlinks with S_ISLNK()).
      
      On Windows with MinGW/MSYS, we need extra code, because the MinGW
      runtime has its own readdir() without d_type, and the lstat()-based
      fallback has no knowledge of our convention for treating junctions as
      symlinks.
      
      Back-patch to 14, where get_dirent_type() landed.
      Reported-by: default avatarAndrew Dunstan <andrew@dunslane.net>
      Discussion: https://postgr.es/m/b9ddf605-6b36-f90d-7c30-7b3e95c46276%40dunslane.net
      5ad478c9
  2. 27 Jul, 2022 1 commit
    • Alvaro Herrera's avatar
      Allow "in place" tablespaces. · 961cab0a
      Alvaro Herrera authored
      This is a backpatch to branches 10-14 of the following commits:
      
      7170f2159fb2 Allow "in place" tablespaces.
      c6f2f01611d4 Fix pg_basebackup with in-place tablespaces.
      f6f0db4d6240 Fix pg_tablespace_location() with in-place tablespaces
      7a7cd84893e0 doc: Remove mention to in-place tablespaces for pg_tablespace_location()
      5344723755bd Remove unnecessary Windows-specific basebackup code.
      
      In-place tablespaces were introduced as a testing helper mechanism, but
      they are going to be used for a bugfix in WAL replay to be backpatched
      to all stable branches.
      
      I (Álvaro) had to adjust some code to account for lack of
      get_dirent_type() in branches prior to 14.
      
      Author: Thomas Munro <thomas.munro@gmail.com>
      Author: Michaël Paquier <michael@paquier.xyz>
      Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
      Discussion: https://postgr.es/m/20220722081858.omhn2in5zt3g4nek@alvherre.pgsql
      961cab0a
  3. 26 Jul, 2022 1 commit
    • Tom Lane's avatar
      Force immediate commit after CREATE DATABASE etc in extended protocol. · 3e1297a6
      Tom Lane authored
      We have a few commands that "can't run in a transaction block",
      meaning that if they complete their processing but then we fail
      to COMMIT, we'll be left with inconsistent on-disk state.
      However, the existing defenses for this are only watertight for
      simple query protocol.  In extended protocol, we didn't commit
      until receiving a Sync message.  Since the client is allowed to
      issue another command instead of Sync, we're in trouble if that
      command fails or is an explicit ROLLBACK.  In any case, sitting
      in an inconsistent state while waiting for a client message
      that might not come seems pretty risky.
      
      This case wasn't reachable via libpq before we introduced pipeline
      mode, but it's always been an intended aspect of extended query
      protocol, and likely there are other clients that could reach it
      before.
      
      To fix, set a flag in PreventInTransactionBlock that tells
      exec_execute_message to force an immediate commit.  This seems
      to be the approach that does least damage to existing working
      cases while still preventing the undesirable outcomes.
      
      While here, add some documentation to protocol.sgml that explicitly
      says how to use pipelining.  That's latent in the existing docs if
      you know what to look for, but it's better to spell it out; and it
      provides a place to document this new behavior.
      
      Per bug #17434 from Yugo Nagata.  It's been wrong for ages,
      so back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/17434-d9f7a064ce2a88a3@postgresql.org
      3e1297a6
  4. 25 Jul, 2022 1 commit
    • Heikki Linnakangas's avatar
      Fix ReadRecentBuffer for local buffers. · 3f968b94
      Heikki Linnakangas authored
      It incorrectly used GetBufferDescriptor instead of
      GetLocalBufferDescriptor, causing it to not find the correct buffer in
      most cases, and performing an out-of-bounds memory read in the corner
      case that temp_buffers > shared_buffers.
      
      It also bumped the usage-count on the buffer, even if it was
      previously pinned. That won't lead to crashes or incorrect results,
      but it's different from what the shared-buffer case does, and
      different from the usual code in LocalBufferAlloc. Fix that too, and
      make the code ordering match LocalBufferAlloc() more closely, so that
      it's easier to verify that it's doing the same thing.
      
      Currently, ReadRecentBuffer() is only used with non-temp relations, in
      WAL redo, so the broken code is currently dead code. However, it could
      be used by extensions.
      
      Backpatch-through: 14
      Discussion: https://www.postgresql.org/message-id/2d74b46f-27c9-fb31-7f99-327a87184cc0%40iki.fi
      Reviewed-by: Thomas Munro, Zhang Mingli, Richard Guo
      3f968b94
  5. 23 Jul, 2022 1 commit
    • Tom Lane's avatar
      Doc: improve documentation about random(). · 31d5354c
      Tom Lane authored
      We didn't explicitly say that random() uses a randomly-chosen seed
      if you haven't called setseed().  Do so.
      
      Also, remove ref/set.sgml's no-longer-accurate (and never very
      relevant) statement that the seed value is multiplied by 2^31-1.
      
      Back-patch to v12 where set.sgml's claim stopped being true.
      The claim that we use a source of random bits as seed was debatable
      before 4203842a, too, so v12 seems like a good place to stop.
      
      Per question from Carl Sopchak.
      
      Discussion: https://postgr.es/m/f37bb937-9d99-08f0-4de7-80c91a3cfc2e@sopchak.me
      31d5354c
  6. 22 Jul, 2022 2 commits
  7. 21 Jul, 2022 3 commits
    • Bruce Momjian's avatar
      doc: use wording "restore" instead of "reload" of dumps · e613466e
      Bruce Momjian authored
      Reported-by: axel.kluener@gmail.com
      
      Discussion: https://postgr.es/m/164736074430.660.3645615289283943146@wrigleys.postgresql.org
      
      Backpatch-through: 11
      e613466e
    • Bruce Momjian's avatar
      doc: clarify that auth. names are lower case and case-sensitive · 21640e98
      Bruce Momjian authored
      This is true even for acronyms that are usually upper case, like LDAP.
      
      Reported-by: Alvaro Herrera
      
      Discussion: https://postgr.es/m/202205141521.2nodjabmsour@alvherre.pgsql
      
      Backpatch-through: 10
      21640e98
    • Tom Lane's avatar
      Fix ruleutils issues with dropped cols in functions-returning-composite. · da9a28fd
      Tom Lane authored
      Due to lack of concern for the case in the dependency code, it's
      possible to drop a column of a composite type even though stored
      queries have references to the dropped column via functions-in-FROM
      that return the composite type.  There are "soft" references,
      namely FROM-clause aliases for such columns, and "hard" references,
      that is actual Vars referring to them.  The right fix for hard
      references is to add dependencies preventing the drop; something
      we've known for many years and not done (and this commit still doesn't
      address it).  A "soft" reference shouldn't prevent a drop though.
      We've been around on this before (cf. 9b35ddce, 2c4debbd), but
      nobody had noticed that the current behavior can result in dump/reload
      failures, because ruleutils.c can print more column aliases than the
      underlying composite type now has.  So we need to rejigger the
      column-alias-handling code to treat such columns as dropped and not
      print aliases for them.
      
      Rather than writing new code for this, I used expandRTE() which already
      knows how to figure out which function result columns are dropped.
      I'd initially thought maybe we could use expandRTE() in all cases, but
      that fails for EXPLAIN's purposes, because the planner strips a lot of
      RTE infrastructure that expandRTE() needs.  So this patch just uses it
      for unplanned function RTEs and otherwise does things the old way.
      
      If there is a hard reference (Var), then removing the column alias
      causes us to fail to print the Var, since there's no longer a name
      to print.  Failing seems less desirable than printing a made-up
      name, so I made it print "?dropped?column?" instead.
      
      Per report from Timo Stolz.  Back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/5c91267e-3b6d-5795-189c-d15a55d61dbb@nullachtvierzehn.de
      da9a28fd
  8. 20 Jul, 2022 2 commits
    • Fujii Masao's avatar
      Fix assertion failure and segmentation fault in backup code. · be2e842c
      Fujii Masao authored
      When a non-exclusive backup is canceled, do_pg_abort_backup() is called
      and resets some variables set by pg_backup_start (pg_start_backup in v14
      or before). But previously it forgot to reset the session state indicating
      whether a non-exclusive backup is in progress or not in this session.
      
      This issue could cause an assertion failure when the session running
      BASE_BACKUP is terminated after it executed pg_backup_start and
      pg_backup_stop (pg_stop_backup in v14 or before). Also it could cause
      a segmentation fault when pg_backup_stop is called after BASE_BACKUP
      in the same session is canceled.
      
      This commit fixes the issue by making do_pg_abort_backup reset
      that session state.
      
      Back-patch to all supported branches.
      
      Author: Fujii Masao
      Reviewed-by: Kyotaro Horiguchi, Masahiko Sawada, Michael Paquier, Robert Haas
      Discussion: https://postgr.es/m/3374718f-9fbf-a950-6d66-d973e027f44c@oss.nttdata.com
      be2e842c
    • Fujii Masao's avatar
      Prevent BASE_BACKUP in the middle of another backup in the same session. · 2aedf25e
      Fujii Masao authored
      Multiple non-exclusive backups are able to be run conrrently in different
      sessions. But, in the same session, only one non-exclusive backup can be
      run at the same moment. If pg_backup_start (pg_start_backup in v14 or before)
      is called in the middle of another non-exclusive backup in the same session,
      an error is thrown.
      
      However, previously, in logical replication walsender mode, even if that
      walsender session had already called pg_backup_start and started
      a non-exclusive backup, it could execute BASE_BACKUP command and
      start another non-exclusive backup. Which caused subsequent pg_backup_stop
      to throw an error because BASE_BACKUP unexpectedly reset the session state
      marked by pg_backup_start.
      
      This commit prevents BASE_BACKUP command in the middle of another
      non-exclusive backup in the same session.
      
      Back-patch to all supported branches.
      
      Author: Fujii Masao
      Reviewed-by: Kyotaro Horiguchi, Masahiko Sawada, Michael Paquier, Robert Haas
      Discussion: https://postgr.es/m/3374718f-9fbf-a950-6d66-d973e027f44c@oss.nttdata.com
      2aedf25e
  9. 18 Jul, 2022 2 commits
  10. 17 Jul, 2022 2 commits
  11. 16 Jul, 2022 1 commit
  12. 15 Jul, 2022 2 commits
  13. 14 Jul, 2022 9 commits
  14. 13 Jul, 2022 1 commit
    • Alvaro Herrera's avatar
      Plug memory leak · 9e038d69
      Alvaro Herrera authored
      Commit 054325c5eeb3 created a memory leak in PQsendQueryInternal in case
      an error occurs while sending the message.  Repair.
      
      Backpatch to 14, like that commit.  Reported by Coverity.
      9e038d69
  15. 12 Jul, 2022 1 commit
    • Tom Lane's avatar
      Invent qsort_interruptible(). · af72b088
      Tom Lane authored
      Justin Pryzby reported that some scenarios could cause gathering
      of extended statistics to spend many seconds in an un-cancelable
      qsort() operation.  To fix, invent qsort_interruptible(), which is
      just like qsort_arg() except that it will also do CHECK_FOR_INTERRUPTS
      every so often.  This bloats the backend by a couple of kB, which
      seems like a good investment.  (We considered just enabling
      CHECK_FOR_INTERRUPTS in the existing qsort and qsort_arg functions,
      but there are some callers for which that'd demonstrably be unsafe.
      Opt-in seems like a better way.)
      
      For now, just apply qsort_interruptible() in statistics collection.
      There's probably more places where it could be useful, but we can
      always change other call sites as we find problems.
      
      Back-patch to v14.  Before that we didn't have extended stats on
      expressions, so that the problem was less severe.  Also, this patch
      depends on the sort_template infrastructure introduced in v14.
      
      Tom Lane and Justin Pryzby
      
      Discussion: https://postgr.es/m/20220509000108.GQ28830@telsasoft.com
      af72b088
  16. 11 Jul, 2022 2 commits
  17. 10 Jul, 2022 1 commit
  18. 09 Jul, 2022 1 commit
  19. 08 Jul, 2022 1 commit
  20. 07 Jul, 2022 1 commit
    • Dean Rasheed's avatar
      Fix alias matching in transformLockingClause(). · 8d846444
      Dean Rasheed authored
      When locking a specific named relation for a FOR [KEY] UPDATE/SHARE
      clause, transformLockingClause() finds the relation to lock by
      scanning the rangetable for an RTE with a matching eref->aliasname.
      However, it failed to account for the visibility rules of a join RTE.
      
      If a join RTE doesn't have a user-supplied alias, it will have a
      generated eref->aliasname of "unnamed_join" that is not visible as a
      relation name in the parse namespace. Such an RTE needs to be skipped,
      otherwise it might be found in preference to a regular base relation
      with a user-supplied alias of "unnamed_join", preventing it from being
      locked.
      
      In addition, if a join RTE doesn't have a user-supplied alias, but
      does have a join_using_alias, then the RTE needs to be matched using
      that alias rather than the generated eref->aliasname, otherwise a
      misleading "relation not found" error will be reported rather than a
      "join cannot be locked" error.
      
      Backpatch all the way, except for the second part which only goes back
      to 14, where JOIN USING aliases were added.
      
      Dean Rasheed, reviewed by Tom Lane.
      
      Discussion: https://postgr.es/m/CAEZATCUY_KOBnqxbTSPf=7fz9HWPnZ5Xgb9SwYzZ8rFXe7nb=w@mail.gmail.com
      8d846444
  21. 05 Jul, 2022 3 commits
    • Tom Lane's avatar
      Tighten pg_upgrade's new check for non-upgradable anyarray usages. · 9783413c
      Tom Lane authored
      We only need to reject cases when the aggregate or operator is
      itself declared with a polymorphic type.  Per buildfarm.
      
      Discussion: https://postgr.es/m/3383880.QJadu78ljV@vejsadalnx
      9783413c
    • Tom Lane's avatar
      Fix pg_upgrade to detect non-upgradable anyarray usages. · 175e60a5
      Tom Lane authored
      When we changed some built-in functions to use anycompatiblearray
      instead of anyarray, we created a dump/restore hazard for user-defined
      operators and aggregates relying on those functions: the user objects
      have to be modified to change their signatures similarly.  This causes
      pg_upgrade to fail partway through if the source installation contains
      such objects.  We generally try to have pg_upgrade detect such hazards
      and fail before it does anything exciting, so add logic to detect
      this case too.
      
      Back-patch to v14 where the change was made.
      
      Justin Pryzby, reviewed by Andrey Borodin
      
      Discussion: https://postgr.es/m/3383880.QJadu78ljV@vejsadalnx
      175e60a5
    • Alvaro Herrera's avatar
      libpq: Improve idle state handling in pipeline mode · 7c1f4261
      Alvaro Herrera authored
      We were going into IDLE state too soon when executing queries via
      PQsendQuery in pipeline mode, causing several scenarios to misbehave in
      different ways -- most notably, as reported by Daniele Varrazzo, that a
      warning message is produced by libpq:
        message type 0x33 arrived from server while idle
      But it is also possible, if queries are sent and results consumed not in
      lockstep, for the expected mediating NULL result values from PQgetResult
      to be lost (a problem which has not been reported, but which is more
      serious).
      
      Fix this by introducing two new concepts: one is a command queue element
      PGQUERY_CLOSE to tell libpq to wait for the CloseComplete server
      response to the Close message that is sent by PQsendQuery.  Because the
      application is not expecting any PGresult from this, the mechanism to
      consume it is a bit hackish.
      
      The other concept, authored by Horiguchi-san, is a PGASYNC_PIPELINE_IDLE
      state for libpq's state machine to differentiate "really idle" from
      merely "the idle state that occurs in between reading results from the
      server for elements in the pipeline".  This makes libpq not go fully
      IDLE when the libpq command queue contains entries; in normal cases, we
      only go IDLE once at the end of the pipeline, when the server response
      to the final SYNC message is received.  (However, there are corner cases
      it doesn't fix, such as terminating the query sequence by
      PQsendFlushRequest instead of PQpipelineSync; this sort of scenario is
      what requires PGQUERY_CLOSE bit above.)
      
      This last bit helps make the libpq state machine clearer; in particular
      we can get rid of an ugly hack in pqParseInput3 to avoid considering
      IDLE as such when the command queue contains entries.
      
      A new test mode is added to libpq_pipeline.c to tickle some related
      problematic cases.
      Reported-by: default avatarDaniele Varrazzo <daniele.varrazzo@gmail.com>
      Co-authored-by: default avatarKyotaro Horiguchi <horikyota.ntt@gmail.com>
      Discussion: https://postgr.es/m/CA+mi_8bvD0_CW3sumgwPvWdNzXY32itoG_16tDYRu_1S2gV2iw@mail.gmail.com
      7c1f4261