1. 05 Feb, 2021 2 commits
  2. 04 Feb, 2021 7 commits
    • Peter Geoghegan's avatar
      Harden nbtree page deletion. · c34787f9
      Peter Geoghegan authored
      Add some additional defensive checks in the second phase of index
      deletion to detect and report index corruption during VACUUM, and to
      avoid having VACUUM become stuck in more cases.  The code is still not
      robust in the presence of a circular chain of sibling links, though it's
      not clear whether that really matters.  This is follow-up work to commit
      3a01f68e.
      
      The new defensive checks rely on the assumption that there can be no
      more than one VACUUM operation running for an index at any given time.
      Remove an old comment suggesting that multiple concurrent VACUUMs need
      to be considered here.  This concern now seems highly unlikely to have
      any real validity, since we clearly rely on the same assumption in
      several other places.  For example, there are much more recent comments
      that appear in the same function (added by commit efada2b8) that make
      the same assumption.
      
      Also add a CHECK_FOR_INTERRUPTS() to the relevant code path.  Contrary
      to comments added by commit 3a01f68e, it is actually possible to handle
      interrupts here, at least in the common case where processing takes
      place at the leaf level.  We only hold a pin on leafbuf/target page when
      stepping right at the leaf level.
      
      No backpatch due to the lack of complaints following hardening added to
      the same area by commit 3a01f68e.
      c34787f9
    • Heikki Linnakangas's avatar
      Fix small error in COPY FROM progress reporting. · 2f86ab30
      Heikki Linnakangas authored
      The # of bytes processed was accumulated slightly incorrectly. After
      loading more data to the input buffer, we added the number of bytes in
      the buffer to the sum. But in case of multi-byte characters or escapes,
      there can be a few unprocessed bytes left over from previous load in the
      buffer. Those bytes got counted twice.
      2f86ab30
    • Peter Eisentraut's avatar
      Refactor Windows error message for easier translation · 3c78e056
      Peter Eisentraut authored
      In the error messages referring to the user right "Lock pages in
      memory", this is a term from the Windows OS, so it should be
      translated in accordance with the OS localization.  Refactor the error
      messages so this is easier and clearer.  Also fix the capitalization
      to match the existing capitalization in the OS.
      3c78e056
    • Michael Paquier's avatar
      Ensure unlinking of old index file with REINDEX (TABLESPACE) · 5128483d
      Michael Paquier authored
      The original versions of the patch included this part, but a mismerge
      from my side has made this piece go missing.  Oversight in c5b28604.
      5128483d
    • Michael Paquier's avatar
      Clarify comment in tablesync.c · fc749bc7
      Michael Paquier authored
      Author: Peter Smith
      Reviewed-by: Amit Kapila, Michael Paquier, Euler Taveira
      Discussion: https://postgr.es/m/CAHut+Pt9_T6pWar0FLtPsygNmme8HPWPdGUyZ_8mE1Yvjdf0ZA@mail.gmail.com
      fc749bc7
    • Michael Paquier's avatar
      Add TABLESPACE option to REINDEX · c5b28604
      Michael Paquier authored
      This patch adds the possibility to move indexes to a new tablespace
      while rebuilding them.  Both the concurrent and the non-concurrent cases
      are supported, and the following set of restrictions apply:
      - When using TABLESPACE with a REINDEX command that targets a
      partitioned table or index, all the indexes of the leaf partitions are
      moved to the new tablespace.  The tablespace references of the non-leaf,
      partitioned tables in pg_class.reltablespace are not changed. This
      requires an extra ALTER TABLE SET TABLESPACE.
      - Any index on a toast table rebuilt as part of a parent table is kept
      in its original tablespace.
      - The operation is forbidden on system catalogs, including trying to
      directly move a toast relation with REINDEX.  This results in an error
      if doing REINDEX on a single object.  REINDEX SCHEMA, DATABASE and
      SYSTEM skip system relations when TABLESPACE is used.
      
      Author: Alexey Kondratov, Michael Paquier, Justin Pryzby
      Reviewed-by: Álvaro Herrera, Michael Paquier
      Discussion: https://postgr.es/m/8a8f5f73-00d3-55f8-7583-1375ca8f6a91@postgrespro.ru
      c5b28604
    • Tom Lane's avatar
      Avoid crash when rolling back within a prepared statement. · 9624321e
      Tom Lane authored
      If a portal is used to run a prepared CALL or DO statement that
      contains a ROLLBACK, PortalRunMulti fails because the portal's
      statement list gets cleared by the rollback.  (Since the grammar
      doesn't allow CALL/DO in PREPARE, the only easy way to get to this is
      via extended query protocol, which treats all inputs as prepared
      statements.)  It's difficult to avoid resetting the portal early
      because of resource-management issues, so work around this by teaching
      PortalRunMulti to be wary of portal->stmts having suddenly become NIL.
      
      The crash has only been seen to occur in v13 and HEAD (as a
      consequence of commit 1cff1b95 having added an extra touch of
      portal->stmts).  But even before that, the code involved touching a
      List that the portal no longer has any claim on.  In the test case at
      hand, the List will still exist because of another refcount on the
      cached plan; but I'm far from convinced that it's impossible for the
      cached plan to have been dropped by the time control gets back to
      PortalRunMulti.  Hence, backpatch to v11 where nested transactions
      were added.
      
      Thomas Munro and Tom Lane, per bug #16811 from James Inform
      
      Discussion: https://postgr.es/m/16811-c1b599b2c6c2d622@postgresql.org
      9624321e
  3. 03 Feb, 2021 3 commits
    • Robert Haas's avatar
      Factor pattern-construction logic out of processSQLNamePattern. · 2c8726c4
      Robert Haas authored
      The logic for converting the shell-glob-like syntax supported by
      utilities like psql and pg_dump to regular expression is
      extracted into a new function patternToSQLRegex. The existing
      function processSQLNamePattern now uses this function as a
      subroutine.
      
      patternToSQLRegex is a little more general than what is required
      by processSQLNamePattern. That function is only interested in
      patterns that can have up to 2 parts, a schema and a relation;
      but patternToSQLRegex can limit the maximum number of parts to
      between 1 and 3, so that patterns can look like either
      "database.schema.relation", "schema.relation", or "relation"
      depending on how it's invoked and what the user specifies.
      
      processSQLNamePattern only passes two buffers, so works exactly
      the same as before, always interpreting the pattern as either
      a "schema.relation" pattern or a "relation" pattern. But,
      future callers can use this function in other ways.
      
      Mark Dilger, reviewed by me. The larger patch series of which this is
      a part has also had review from Peter Geoghegan, Andres Freund, Álvaro
      Herrera, Michael Paquier, and Amul Sul, but I don't know whether
      any of them have reviewed this bit specifically.
      
      Discussion: http://postgr.es/m/12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com
      Discussion: http://postgr.es/m/5F743835-3399-419C-8324-2D424237E999@enterprisedb.com
      Discussion: http://postgr.es/m/70655DF3-33CE-4527-9A4D-DDEB582B6BA0@enterprisedb.com
      2c8726c4
    • Tom Lane's avatar
      Remove special BKI_LOOKUP magic for namespace and role OIDs. · ba0faf81
      Tom Lane authored
      Now that commit 62f34097 attached BKI_LOOKUP annotation to all the
      namespace and role OID columns in the catalogs, there's no real reason
      to have the magic PGNSP and PGUID symbols.  Get rid of them in favor
      of implementing those lookups according to genbki.pl's normal pattern.
      
      This means that in the catalog headers, BKI_DEFAULT(PGNSP) becomes
      BKI_DEFAULT(pg_catalog), which seems a lot more transparent.
      BKI_DEFAULT(PGUID) becomes BKI_DEFAULT(POSTGRES), which is perhaps
      less so; but you can look into pg_authid.dat to discover that
      POSTGRES is the nonce name for the bootstrap superuser.
      
      This change also means that if we ever need cross-references in the
      initial catalog data to any of the other built-in roles besides
      POSTGRES, or to some other built-in schema besides pg_catalog,
      we can just do it.
      
      No catversion bump here, as there's no actual change in the contents
      of postgres.bki.
      
      Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us
      ba0faf81
    • Peter Eisentraut's avatar
      pg_dump: Fix dumping of inherited generated columns · 0bf83648
      Peter Eisentraut authored
      Generation expressions of generated columns are always inherited, so
      there is no need to set them separately in child tables, and there is
      no syntax to do so either.  The code previously used the code paths
      for the handling of default values, for which different rules apply;
      in particular it might want to set a default value explicitly for an
      inherited column.  This resulted in unrestorable dumps.  For generated
      columns, just skip them in inherited tables.
      Reviewed-by: default avatarTom Lane <tgl@sss.pgh.pa.us>
      Discussion: https://www.postgresql.org/message-id/flat/15830.1575468847%40sss.pgh.pa.us
      0bf83648
  4. 02 Feb, 2021 7 commits
    • Tom Lane's avatar
      Retire findoidjoins. · ef3d4613
      Tom Lane authored
      In the wake of commit 62f34097, we no longer need this tool.
      
      Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us
      ef3d4613
    • Tom Lane's avatar
      Build in some knowledge about foreign-key relationships in the catalogs. · 62f34097
      Tom Lane authored
      This follows in the spirit of commit dfb75e47, which created primary
      key and uniqueness constraints to improve the visibility of constraints
      imposed on the system catalogs.  While our catalogs contain many
      foreign-key-like relationships, they don't quite follow SQL semantics,
      in that the convention for an omitted reference is to write zero not
      NULL.  Plus, we have some cases in which there are arrays each of whose
      elements is supposed to be an FK reference; SQL has no way to model that.
      So we can't create actual foreign key constraints to describe the
      situation.  Nonetheless, we can collect and use knowledge about these
      relationships.
      
      This patch therefore adds annotations to the catalog header files to
      declare foreign-key relationships.  (The BKI_LOOKUP annotations cover
      simple cases, but we weren't previously distinguishing which such
      columns are allowed to contain zeroes; we also need new markings for
      multi-column FK references.)  Then, Catalog.pm and genbki.pl are
      taught to collect this information into a table in a new generated
      header "system_fk_info.h".  The only user of that at the moment is
      a new SQL function pg_get_catalog_foreign_keys(), which exposes the
      table to SQL.  The oidjoins regression test is rewritten to use
      pg_get_catalog_foreign_keys() to find out which columns to check.
      Aside from removing the need for manual maintenance of that test
      script, this allows it to cover numerous relationships that were not
      checked by the old implementation based on findoidjoins.  (As of this
      commit, 217 relationships are checked by the test, versus 181 before.)
      
      Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us
      62f34097
    • Tom Lane's avatar
      Doc: consistently identify OID catalog columns that can be zero. · 47933140
      Tom Lane authored
      Not all OID-reference columns that can contain zeroes (indicating
      "no reference") were explicitly marked in catalogs.sgml.  Fix that,
      and try to make the style a little more consistent while at it ---
      for example, consistently write "zero" not "0" for these cases.
      
      Joel Jacobson and Tom Lane
      
      Discussion: https://postgr.es/m/4ed9a372-7bf9-479a-926c-ae8e774717a8@www.fastmail.com
      47933140
    • Tom Lane's avatar
      Remove extra increment of plpgsql's statement counter for FOR loops. · dfcc46fe
      Tom Lane authored
      This left gaps in the internal statement numbering, which is not
      terribly harmful (else we'd have noticed sooner), but it's not
      great either.
      
      Oversight in bbd5c207; backpatch to v12 where that came in.
      
      Pavel Stehule
      
      Discussion: https://postgr.es/m/CAFj8pRDXyQaJmpotNTQVc-t-WxdWZC35V2PnmwOaV1-taidFWA@mail.gmail.com
      dfcc46fe
    • Tom Lane's avatar
      Fix ancient memory leak in contrib/auto_explain. · 5c0f7cc5
      Tom Lane authored
      The ExecutorEnd hook is invoked in a context that could be quite
      long-lived, not the executor's own per-query context as I think
      we were sort of assuming.  Thus, any cruft generated while producing
      the EXPLAIN output could accumulate over multiple queries.  This can
      result in spectacular leakage if log_nested_statements is on, and
      even without that I'm surprised nobody complained before.
      
      To fix, just switch into the executor's context so that anything we
      allocate will be released when standard_ExecutorEnd frees the executor
      state.  We might as well nuke the code's retail pfree of the explain
      output string, too; that's laughably inadequate to the need.
      
      Japin Li, per report from Jeff Janes.  This bug is old, so
      back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/CAMkU=1wCVtbeRn0s9gt12KwQ7PLXovbpM8eg25SYocKW3BT4hg@mail.gmail.com
      5c0f7cc5
    • Peter Eisentraut's avatar
      Improve confusing variable names · 1d71f3c8
      Peter Eisentraut authored
      The prototype calls the second argument of
      pgstat_progress_update_multi_param() "index", and some callers name
      their local variable that way.  But when the surrounding code deals
      with index relations, this is confusing, and in at least one case
      shadowed another variable that is referring to an index relation.
      Adjust those call sites to have clearer local variable naming, similar
      to existing callers in indexcmds.c.
      1d71f3c8
    • Michael Paquier's avatar
      Remove unused column atttypmod from initial tablesync query · 4ad31bb2
      Michael Paquier authored
      The initial tablesync done by logical replication used a query to fetch
      the information of a relation's columns that included atttypmod, but it
      was left unused.  This was added by 7c4f5240.
      
      Author: Euler Taveira
      Reviewed-by: Önder Kalacı, Amit Langote, Japin Li
      Discussion: https://postgr.es/m/CAHE3wggb715X+mK_DitLXF25B=jE6xyNCH4YOwM860JR7HarGQ@mail.gmail.com
      4ad31bb2
  5. 01 Feb, 2021 8 commits
  6. 31 Jan, 2021 5 commits
  7. 30 Jan, 2021 7 commits
    • Peter Eisentraut's avatar
      Add primary keys and unique constraints to system catalogs · dfb75e47
      Peter Eisentraut authored
      For those system catalogs that have a unique indexes, make a primary
      key and unique constraint, using ALTER TABLE ... PRIMARY KEY/UNIQUE
      USING INDEX.
      
      This can be helpful for GUI tools that look for a primary key, and it
      might in the future allow declaring foreign keys, for making schema
      diagrams.
      
      The constraint creation statements are automatically created by
      genbki.pl from DECLARE_UNIQUE_INDEX directives.  To specify which one
      of the available unique indexes is the primary key, use the new
      directive DECLARE_UNIQUE_INDEX_PKEY instead.  By convention, we
      usually make a catalog's OID column its primary key, if it has one.
      Reviewed-by: default avatarTom Lane <tgl@sss.pgh.pa.us>
      Discussion: https://www.postgresql.org/message-id/flat/dc5f44d9-5ec1-a596-0251-dadadcdede98@2ndquadrant.com
      dfb75e47
    • Peter Eisentraut's avatar
      doc: Clarify status of SELECT INTO on reference page · 65330622
      Peter Eisentraut authored
      The documentation as well as source code comments weren't entirely
      clear whether SELECT INTO was truly deprecated (thus in theory
      destined to be removed eventually), or just a less recommended
      variant.  After discussion, it appears that other implementations also
      use SELECT INTO in direct SQL in a way similar to PostgreSQL, so it
      seems worth keeping it for compatibility.  Update the language in the
      documentation to that effect.
      
      Discussion: https://www.postgresql.org/message-id/flat/96dc0df3-e13a-a85d-d045-d6e2c85218da%40enterprisedb.com
      65330622
    • Peter Eisentraut's avatar
      Allow GRANTED BY clause in normal GRANT and REVOKE statements · 6aaaa76b
      Peter Eisentraut authored
      The SQL standard allows a GRANTED BY clause on GRANT and
      REVOKE (privilege) statements that can specify CURRENT_USER or
      CURRENT_ROLE.  In PostgreSQL, both of these are the default behavior.
      Since we already have all the parsing support for this for the
      GRANT (role) statement, we might as well add basic support for this
      for the privilege variant as well.  This allows us to check off SQL
      feature T332.  In the future, perhaps more interesting things could be
      done with this, too.
      Reviewed-by: default avatarSimon Riggs <simon@2ndquadrant.com>
      Discussion: https://www.postgresql.org/message-id/flat/f2feac44-b4c5-f38f-3699-2851d6a76dc9@2ndquadrant.com
      6aaaa76b
    • Noah Misch's avatar
      Revive "snapshot too old" with wal_level=minimal and SET TABLESPACE. · 7da83415
      Noah Misch authored
      Given a permanent relation rewritten in the current transaction, the
      old_snapshot_threshold mechanism assumed the relation had never been
      subject to early pruning.  Hence, a query could fail to report "snapshot
      too old" when the rewrite followed an early truncation.  ALTER TABLE SET
      TABLESPACE is probably the only rewrite mechanism capable of exposing
      this bug.  REINDEX sets indcheckxmin, avoiding the problem.  CLUSTER has
      zeroed page LSNs since before old_snapshot_threshold existed, so
      old_snapshot_threshold has never cooperated with it.  ALTER TABLE
      ... SET DATA TYPE makes the table look empty to every past snapshot,
      which is strictly worse.  Back-patch to v13, where commit
      c6b92041 broke this.
      
      Kyotaro Horiguchi and Noah Misch
      
      Discussion: https://postgr.es/m/20210113.160705.2225256954956139776.horikyota.ntt@gmail.com
      7da83415
    • Noah Misch's avatar
      Fix error with CREATE PUBLICATION, wal_level=minimal, and new tables. · 360bd232
      Noah Misch authored
      CREATE PUBLICATION has failed spuriously when applied to a permanent
      relation created or rewritten in the current transaction.  Make the same
      change to another site having the same semantic intent; the second
      instance has no user-visible consequences.  Back-patch to v13, where
      commit c6b92041 broke this.
      
      Kyotaro Horiguchi
      
      Discussion: https://postgr.es/m/20210113.160705.2225256954956139776.horikyota.ntt@gmail.com
      360bd232
    • Noah Misch's avatar
      Fix CREATE INDEX CONCURRENTLY for simultaneous prepared transactions. · 8a54e12a
      Noah Misch authored
      In a cluster having used CREATE INDEX CONCURRENTLY while having enabled
      prepared transactions, queries that use the resulting index can silently
      fail to find rows.  Fix this for future CREATE INDEX CONCURRENTLY by
      making it wait for prepared transactions like it waits for ordinary
      transactions.  This expands the VirtualTransactionId structure domain to
      admit prepared transactions.  It may be necessary to reindex to recover
      from past occurrences.  Back-patch to 9.5 (all supported versions).
      
      Andrey Borodin, reviewed (in earlier versions) by Tom Lane and Michael
      Paquier.
      
      Discussion: https://postgr.es/m/2E712143-97F7-4890-B470-4A35142ABC82@yandex-team.ru
      8a54e12a
    • Fujii Masao's avatar
      postgres_fdw: Fix tests for CLOBBER_CACHE_ALWAYS. · f77717b2
      Fujii Masao authored
      The regression tests added in commits 708d165d and 411ae649 caused
      buildfarm failures when  CLOBBER_CACHE_ALWAYS was enabled.
      This commit stabilizes those tests.
      
      The foreign server connections established by postgres_fdw behaves
      differently depending on whether CLOBBER_CACHE_ALWAYS is enabled or not.
      If it's not enabled, those connections are cached. On the other hand,
      if it's enabled, when the connections are established outside transaction
      block, they are not cached (i.e., they are immediately closed at the end of
      query that established them). So the subsequent postgres_fdw_get_connections()
      cannot list those connections and postgres_fdw_disconnect() cannot close them
      (because they are already closed).
      
      When the connections are established inside transaction block, they are
      cached whether CLOBBER_CACHE_ALWAYS was enabled or not. But if it's enabled,
      they are immediately marked as invalid, otherwise not. This causes the
      subsequent postgres_fdw_get_connections() to return different result in
      "valid" column depending on whether CLOBBER_CACHE_ALWAYS was enabled or not.
      
      This commit prevents the above differences of behavior from
      affecting the regression tests.
      
      Per buildfarm failure on trilobite.
      
      Original patch by Bharath Rupireddy. I (Fujii Masao) extracted
      the regression test fix from that and revised it a bit.
      
      Reported-by: Tom Lane
      Author: Bharath Rupireddy
      Reviewed-by: Fujii Masao
      Discussion: https://postgr.es/m/2688508.1611865371@sss.pgh.pa.us
      f77717b2
  8. 29 Jan, 2021 1 commit