1. 02 Dec, 2020 6 commits
    • Stephen Frost's avatar
      Add GSS information to connection authorized log message · dc11f31a
      Stephen Frost authored
      GSS information (if used) such as if the connection was authorized using
      GSS or if it was encrypted using GSS, and perhaps most importantly, what
      the GSS principal used for the authentication was, is extremely useful
      but wasn't being included in the connection authorized log message.
      
      Therefore, add to the connection authorized log message that
      information, in a similar manner to how we log SSL information when SSL
      is used for a connection.
      
      Author: Vignesh C
      Reviewed-by: Bharath Rupireddy
      Discussion: https://www.postgresql.org/message-id/CALDaNm2N1385_Ltoo%3DS7VGT-ESu_bRQa-sC1wg6ikrM2L2Z49w%40mail.gmail.com
      dc11f31a
    • Fujii Masao's avatar
      Track total number of WAL records, FPIs and bytes generated in the cluster. · 01469241
      Fujii Masao authored
      Commit 6b466bf5 allowed pg_stat_statements to track the number of
      WAL records, full page images and bytes that each statement generated.
      Similarly this commit allows us to track the cluster-wide WAL statistics
      counters.
      
      New columns wal_records, wal_fpi and wal_bytes are added into the
      pg_stat_wal view, and reports the total number of WAL records,
      full page images and bytes generated in the , respectively.
      
      Author: Masahiro Ikeda
      Reviewed-by: Amit Kapila, Movead Li, Kyotaro Horiguchi, Fujii Masao
      Discussion: https://postgr.es/m/35ef960128b90bfae3b3fdf60a3a860f@oss.nttdata.com
      01469241
    • Michael Paquier's avatar
      Fix compilation warnings in cryptohash_openssl.c · 91624c2f
      Michael Paquier authored
      These showed up with -O2.  Oversight in 87ae9691.
      
      Author: Fujii Masao
      Discussion: https://postgr.es/m/cee3df00-566a-400c-1252-67c3701f918a@oss.nttdata.com
      91624c2f
    • Fujii Masao's avatar
      Allow restore_command parameter to be changed with reload. · 942305a3
      Fujii Masao authored
      This commit changes restore_command from PGC_POSTMASTER to PGC_SIGHUP.
      
      As the side effect of this commit, restore_command can be reset to
      empty during archive recovery. In this setting, archive recovery
      tries to replay only WAL files available in pg_wal directory. This is
      the same behavior as when the command that always fails is specified
      in restore_command.
      
      Note that restore_command still must be specified (not empty) when
      starting archive recovery, even after applying this commit. This is
      necessary as the safeguard to prevent users from forgetting to
      specify restore_command and starting archive recovery.
      
      Thanks to Peter Eisentraut, Michael Paquier, Andres Freund,
      Robert Haas and Anastasia Lubennikova for discussion.
      
      Author: Sergei Kornilov
      Reviewed-by: Kyotaro Horiguchi, Fujii Masao
      Discussion: https://postgr.es/m/2317771549527294@sas2-985f744271ca.qloud-c.yandex.net
      942305a3
    • Michael Paquier's avatar
      Move SHA2 routines to a new generic API layer for crypto hashes · 87ae9691
      Michael Paquier authored
      Two new routines to allocate a hash context and to free it are created,
      as these become necessary for the goal behind this refactoring: switch
      the all cryptohash implementations for OpenSSL to use EVP (for FIPS and
      also because upstream does not recommend the use of low-level cryptohash
      functions for 20 years).  Note that OpenSSL hides the internals of
      cryptohash contexts since 1.1.0, so it is necessary to leave the
      allocation to OpenSSL itself, explaining the need for those two new
      routines.  This part is going to require more work to properly track
      hash contexts with resource owners, but this not introduced here.
      Still, this refactoring makes the move possible.
      
      This reduces the number of routines for all SHA2 implementations from
      twelve (SHA{224,256,386,512} with init, update and final calls) to five
      (create, free, init, update and final calls) by incorporating the hash
      type directly into the hash context data.
      
      The new cryptohash routines are moved to a new file, called cryptohash.c
      for the fallback implementations, with SHA2 specifics becoming a part
      internal to src/common/.  OpenSSL specifics are part of
      cryptohash_openssl.c.  This infrastructure is usable for more hash
      types, like MD5 or HMAC.
      
      Any code paths using the internal SHA2 routines are adapted to report
      correctly errors, which are most of the changes of this commit.  The
      zones mostly impacted are checksum manifests, libpq and SCRAM.
      
      Note that e21cbb4b was a first attempt to switch SHA2 to EVP, but it
      lacked the refactoring needed for libpq, as done here.
      
      This patch has been tested on Linux and Windows, with and without
      OpenSSL, and down to 1.0.1, the oldest version supported on HEAD.
      
      Author: Michael Paquier
      Reviewed-by: Daniel Gustafsson
      Discussion: https://postgr.es/m/20200924025314.GE7405@paquier.xyz
      87ae9691
    • Bruce Momjian's avatar
      pg_checksums: data_checksum_version is unsigned so use %u not %d · 888671a8
      Bruce Momjian authored
      While the previous behavior didn't generate a warning, we might as well
      use an accurate *printf specification.
      
      Backpatch-through: 12
      888671a8
  2. 01 Dec, 2020 7 commits
  3. 30 Nov, 2020 9 commits
  4. 29 Nov, 2020 3 commits
    • Tom Lane's avatar
      Fix recently-introduced breakage in psql's \connect command. · 7e5e1bba
      Tom Lane authored
      Through my misreading of what the existing code actually did,
      commits 85c54287 et al. broke psql's behavior for the case where
      "\c connstring" provides a password in the connstring.  We should
      use that password in such a case, but as of 85c54287 we ignored it
      (and instead, prompted for a password).
      
      Commit 94929f1c fixed that in HEAD, but since I thought it was
      cleaning up a longstanding misbehavior and not one I'd just created,
      I didn't back-patch it.
      
      Hence, back-patch the portions of 94929f1c having to do with
      password management.  In addition to fixing the introduced bug,
      this means that "\c -reuse-previous=on connstring" will allow
      re-use of an existing connection's password if the connstring
      doesn't change user/host/port.  That didn't happen before, but
      it seems like a bug fix, and anyway I'm loath to have significant
      differences in this code across versions.
      
      Also fix an error with the same root cause about whether or not to
      override a connstring's setting of client_encoding.  As of 85c54287
      we always did so; restore the previous behavior of overriding only
      when stdin/stdout are a terminal and there's no environment setting
      of PGCLIENTENCODING.  (I find that definition a bit surprising, but
      right now doesn't seem like the time to revisit it.)
      
      Per bug #16746 from Krzysztof Gradek.  As with the previous patch,
      back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/16746-44b30e2edf4335d4@postgresql.org
      7e5e1bba
    • Tom Lane's avatar
      Doc: clarify behavior of PQconnectdbParams(). · d5e2bdf7
      Tom Lane authored
      The documentation omitted the critical tidbit that a keyword-array entry
      is simply ignored if its corresponding value-array entry is NULL or an
      empty string; it will *not* override any previously-obtained value for
      the parameter.  (See conninfo_array_parse().)  I'd supposed that would
      force the setting back to default, which is what led me into bug #16746;
      but it doesn't.
      
      While here, I couldn't resist the temptation to do some copy-editing,
      both in the description of PQconnectdbParams() and in the section
      about connection URI syntax.
      
      Discussion: https://postgr.es/m/931505.1606618746@sss.pgh.pa.us
      d5e2bdf7
    • Noah Misch's avatar
      Retry initial slurp_file("current_logfiles"), in test 004_logrotate.pl. · 0f89ca08
      Noah Misch authored
      Buildfarm member topminnow failed when the test script attempted this
      before the syslogger would have created the file.  Back-patch to v12,
      which introduced the test.
      0f89ca08
  5. 28 Nov, 2020 2 commits
    • Tom Lane's avatar
      Clean up after tests in src/test/locale/. · b90a7fe1
      Tom Lane authored
      Oversight in 257836a7, which added these tests.
      b90a7fe1
    • Tom Lane's avatar
      Fix a recently-introduced race condition in LISTEN/NOTIFY handling. · 9c83b54a
      Tom Lane authored
      Commit 566372b3 fixed some race conditions involving concurrent
      SimpleLruTruncate calls, but it introduced new ones in async.c.
      A newly-listening backend could attempt to read Notify SLRU pages that
      were in process of being truncated, possibly causing an error.  Also,
      the QUEUE_TAIL pointer could become set to a value that's not equal to
      the queue position of any backend.  While that's fairly harmless in
      v13 and up (thanks to commit 51004c71), in older branches it resulted
      in near-permanent disabling of the queue truncation logic, so that
      continued use of NOTIFY led to queue-fill warnings and eventual
      inability to send any more notifies.  (A server restart is enough to
      make that go away, but it's still pretty unpleasant.)
      
      The core of the problem is confusion about whether QUEUE_TAIL
      represents the "logical" tail of the queue (i.e., the oldest
      still-interesting data) or the "physical" tail (the oldest data we've
      not yet truncated away).  To fix, split that into two variables.
      QUEUE_TAIL regains its definition as the logical tail, and we
      introduce a new variable to track the oldest un-truncated page.
      
      Per report from Mikael Gustavsson.  Like the previous patch,
      back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/1b8561412e8a4f038d7a491c8b922788@smhi.se
      9c83b54a
  6. 27 Nov, 2020 4 commits
    • Fujii Masao's avatar
      Fix CLUSTER progress reporting of number of blocks scanned. · 3df51ca8
      Fujii Masao authored
      Previously pg_stat_progress_cluster view reported the current block
      number in heap scan as the number of heap blocks scanned (i.e.,
      heap_blks_scanned). This reported number could be incorrect when
      synchronize_seqscans is enabled, because it allowed the heap scan to
      start at block in middle. This could result in wraparounds in the
      heap_blks_scanned column when the heap scan wrapped around.
      This commit fixes the bug by calculating the number of blocks from
      the block that the heap scan starts at to the current block in scan,
      and reporting that number in the heap_blks_scanned column.
      
      Also, in pg_stat_progress_cluster view, previously heap_blks_scanned
      could not reach heap_blks_total at the end of heap scan phase
      if the last pages scanned were empty. This commit fixes the bug by
      manually updating heap_blks_scanned to the same value as
      heap_blks_total when the heap scan phase finishes.
      
      Back-patch to v12 where pg_stat_progress_cluster view was introduced.
      
      Reported-by: Matthias van de Meent
      Author: Matthias van de Meent
      Reviewed-by: Fujii Masao
      Discussion: https://postgr.es/m/CAEze2WjCBWSGkVfYag001Rc4+-nNLDpWM7QbyD6yPvuhKs-gYQ@mail.gmail.com
      3df51ca8
    • Fujii Masao's avatar
      Use standard SIGTERM signal handler die() in test_shm_mq worker. · ef848f4a
      Fujii Masao authored
      Previously test_shm_mq worker used the stripped-down version of die()
      as the SIGTERM signal handler. This commit makes it use die(), instead,
      to simplify the code.
      
      In terms of the code, the difference between die() and the stripped-down
      version previously used is whether the signal handler directly may call
      ProcessInterrupts() or not. But this difference doesn't exist in
      a background worker because, in bgworker, DoingCommandRead flag will
      never be true and die() will never call ProcessInterrupts() directly.
      Therefore test_shm_mq worker can safely use die(), like other bgworker
      proceses (e.g., logical replication apply launcher or autoprewarm worker)
      currently do.
      
      Thanks to Craig Ringer for the report and investigation of the issue.
      
      Author: Bharath Rupireddy
      Reviewed-by: Fujii Masao
      Discussion: https://postgr.es/m/CAGRY4nxsAe_1k_9g5b47orA0S011iBoHsXHFMH7cg7HV0O1bwQ@mail.gmail.com
      ef848f4a
    • Fujii Masao's avatar
      Use standard SIGHUP and SIGTERM signal handlers in worker_spi. · 2a084772
      Fujii Masao authored
      Previously worker_spi used its custom signal handlers for SIGHUP and
      SIGTERM. This commit makes worker_spi use the standard signal handlers,
      to simplify the code.
      
      Note that die() is used as the standard SIGTERM signal handler in
      worker_spi instead of SignalHandlerForShutdownRequest() or bgworker_die().
      Previously the exit handling was only able to exit from within the main loop,
      and not from within the backend code it calls. This is why die() needs to
      be used here, so worker_spi can respond to SIGTERM signal while it's
      executing a query.
      
      Maybe we can say that it's a bug that worker_spi could not respond to
      SIGTERM during query execution. But since worker_spi is a just example
      of the background worker code, we don't do the back-patch.
      
      Thanks to Craig Ringer for the report and investigation of the issue.
      
      Author: Bharath Rupireddy
      Reviewed-by: Fujii Masao
      Discussion: https://postgr.es/m/CALj2ACXDEZhAFOTDcqO9cFSRvrFLyYOnPKrcA1UG4uZn9hUAVg@mail.gmail.com
      Discussion: https://postgr.es/m/CAGRY4nxsAe_1k_9g5b47orA0S011iBoHsXHFMH7cg7HV0O1bwQ@mail.gmail.com
      2a084772
    • Amit Kapila's avatar
      Fix replication of in-progress transactions in tablesync worker. · 0926e96c
      Amit Kapila authored
      Tablesync worker runs under a single transaction but in streaming mode, we
      were committing the transaction on stream_stop, stream_abort, and
      stream_commit. We need to avoid committing the transaction in a streaming
      mode in tablesync worker.
      
      In passing move the call to process_syncing_tables in
      apply_handle_stream_commit after clean up of stream files. This will
      allow clean up of files to happen before the exit of tablesync worker
      which would otherwise be handled by one of the proc exit routines.
      
      Author: Dilip Kumar
      Reviewed-by: Amit Kapila and Peter Smith
      Tested-by: Peter Smith
      Discussion: https://postgr.es/m/CAHut+Pt4PyKQCwqzQ=EFF=bpKKJD7XKt_S23F6L20ayQNxg77A@mail.gmail.com
      0926e96c
  7. 26 Nov, 2020 4 commits
  8. 25 Nov, 2020 5 commits
    • Alvaro Herrera's avatar
      Avoid spurious waits in concurrent indexing · c98763bf
      Alvaro Herrera authored
      In the various waiting phases of CREATE INDEX CONCURRENTLY (CIC) and
      REINDEX CONCURRENTLY (RC), we wait for other processes to release their
      snapshots; this is necessary in general for correctness.  However,
      processes doing CIC in other tables cannot possibly affect CIC or RC
      done in "this" table, so we don't need to wait for those.  This commit
      adds a flag in MyProc->statusFlags to indicate that the current process
      is doing CIC, so that other processes doing CIC or RC can ignore it when
      waiting.
      
      Note that this logic is only valid if the index does not access other
      tables.  For simplicity we avoid setting the flag if the index has a
      column that's an expression, or has a WHERE predicate.  (It is possible
      to have expressional or partial indexes that do not access other tables,
      but figuring that out would require more work.)
      
      This flag can potentially also be used by processes doing REINDEX
      CONCURRENTLY to be skipped; and by VACUUM to ignore processes in CIC or
      RC for the purposes of computing an Xmin.  That's left for future
      commits.
      
      Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
      Author: Dimitry Dolgov <9erthalion6@gmail.com>
      Reviewed-by: default avatarMichael Paquier <michael@paquier.xyz>
      Discussion: https://postgr.es/m/20200810233815.GA18970@alvherre.pgsql
      c98763bf
    • Tom Lane's avatar
      In psql's \d commands, don't truncate attribute default values. · 314fb9ba
      Tom Lane authored
      Historically, psql has truncated the text of a column's default
      expression at 128 characters.  This is unlike any other behavior
      in describe.c, and it's become particularly confusing now that
      the limit is only applied to the expression proper and not to
      the "generated always as (...) stored" text that may get wrapped
      around it.
      
      Excavation in our git history suggests that the original motivation
      for this limit was not really to limit the display width (as I'd long
      supposed), but to make it safe to use a fixed-width output buffer to
      store the result.  That implementation restriction is long gone of
      course, but the limit remained.  Let's just get rid of it.
      
      While here, rearrange the logic about when to free the output string
      so that it's not so dependent on unstated assumptions about the
      possible values of attidentity and attgenerated.
      
      Per bug #16743 from David Turon.  Back-patch to v12 where GENERATED
      came in.  (Arguably we could take it back further, but I'm hesitant
      to change the behavior of long-stable branches for this.)
      
      Discussion: https://postgr.es/m/16743-7b1bacc4af76e7ad@postgresql.org
      314fb9ba
    • Tom Lane's avatar
      Doc: minor improvements for section 11.2 "Index Types". · 85b4ba73
      Tom Lane authored
      Break the per-index-type discussions into <sect2>'s so as to make
      them more visually separate and easier to find.  Improve the markup,
      and make a couple of small wording adjustments.
      
      This also fixes one stray reference to the now-deprecated point
      operators <^ and >^.
      
      Dagfinn Ilmari Mannsåker, reviewed by David Johnston and Jürgen Purtz
      
      Discussion: https://postgr.es/m/877dukhvzg.fsf@wibble.ilmari.org
      85b4ba73
    • Tom Lane's avatar
      Avoid spamming the client with multiple ParameterStatus messages. · 2432b1a0
      Tom Lane authored
      Up to now, we sent a ParameterStatus message to the client immediately
      upon any change in the active value of any GUC_REPORT variable.  This
      was only barely okay when the feature was designed; now that we have
      things like function SET clauses, there are very plausible use-cases
      where a GUC_REPORT variable might change many times within a query
      --- and even end up back at its original value, perhaps.  Fortunately
      most of our GUC_REPORT variables are unlikely to be changed often;
      but there are proposals in play to enlarge that set, or even make it
      user-configurable.
      
      Hence, let's fix things to not generate more than one ParameterStatus
      message per variable per query, and to not send any message at all
      unless the end-of-query value is different from what we last reported.
      
      Discussion: https://postgr.es/m/5708.1601145259@sss.pgh.pa.us
      2432b1a0
    • Peter Eisentraut's avatar
      tablefunc: Reject negative number of tuples passed to normal_rand() · f7399926
      Peter Eisentraut authored
      The function converted the first argument i.e. the number of tuples to
      return into an unsigned integer which turns out to be huge number when
      a negative value is passed.  This causes the function to take much
      longer time to execute.  Instead, reject a negative value.
      
      (If someone really wants to generate many more result rows, they
      should consider adding a bigint or numeric variant.)
      
      While at it, improve SQL test to test the number of tuples returned by
      this function.
      
      Author: Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com>
      Discussion: https://www.postgresql.org/message-id/CAG-ACPW3PUUmSnM6cLa9Rw4BEC5cEMKjX8Gogc8gvQcT3cYA1A@mail.gmail.com
      f7399926