1. 29 Sep, 2020 7 commits
    • Tom Lane's avatar
      Fix memory leak in plpgsql's CALL processing. · a6b1f536
      Tom Lane authored
      When executing a CALL or DO in a non-atomic context (i.e., not inside
      a function or query), plpgsql creates a new plan each time through,
      as a rather hacky solution to some resource management issues.  But
      it failed to free this plan until exit of the current procedure or DO
      block, resulting in serious memory bloat in procedures that called
      other procedures many times.  Fix by remembering to free the plan,
      and by being more honest about restoring the previous state (otherwise,
      recursive procedure calls have a problem).
      
      There was also a smaller leak associated with recalculation of the
      "target" list of output variables.  Fix that by using the statement-
      lifespan context to hold non-permanent values.
      
      Back-patch to v11 where procedures were introduced.
      
      Pavel Stehule and Tom Lane
      
      Discussion: https://postgr.es/m/CAFj8pRDiiU1dqym+_P4_GuTWm76knJu7z9opWayBJTC0nQGUUA@mail.gmail.com
      a6b1f536
    • Alexander Korotkov's avatar
      Support for ISO 8601 in the jsonpath .datetime() method · 927d9abb
      Alexander Korotkov authored
      The SQL standard doesn't require jsonpath .datetime() method to support the
      ISO 8601 format.  But our to_json[b]() functions convert timestamps to text in
      the ISO 8601 format in the sake of compatibility with javascript.  So, we add
      support of the  ISO 8601 to the jsonpath .datetime() in the sake compatibility
      with to_json[b]().
      
      The standard mode of datetime parsing currently supports just template patterns
      and separators in the format string.  In order to implement ISO 8601, we have to
      add support of the format string double quotes to the standard parsing mode.
      
      Discussion: https://postgr.es/m/94321be0-cc96-1a81-b6df-796f437f7c66%40postgrespro.ru
      Author: Nikita Glukhov, revised by me
      Backpatch-through: 13
      927d9abb
    • Alexander Korotkov's avatar
      Remove excess space from jsonpath .datetime() default format string · c2aa562e
      Alexander Korotkov authored
      bffe1bd6 has introduced jsonpath .datetime() method, but default formats
      for time and timestamp contain excess space between time and timezone.  This
      commit removes this excess space making behavior of .datetime() method
      standard-compliant.
      
      Discussion: https://postgr.es/m/94321be0-cc96-1a81-b6df-796f437f7c66%40postgrespro.ru
      Author: Nikita Glukhov
      Backpatch-through: 13
      c2aa562e
    • Fujii Masao's avatar
      Archive timeline history files in standby if archive_mode is set to "always". · fd26f782
      Fujii Masao authored
      Previously the standby server didn't archive timeline history files
      streamed from the primary even when archive_mode is set to "always",
      while it archives the streamed WAL files. This could cause the PITR to
      fail because there was no required timeline history file in the archive.
      The cause of this issue was that walreceiver didn't mark those files as
      ready for archiving.
      
      This commit makes walreceiver mark those streamed timeline history
      files as ready for archiving if archive_mode=always. Then the archiver
      process archives the marked timeline history files.
      
      Back-patch to all supported versions.
      
      Reported-by: Grigory Smolkin
      Author: Grigory Smolkin, Fujii Masao
      Reviewed-by: David Zhang, Anastasia Lubennikova
      Discussion: https://postgr.es/m/54b059d4-2b48-13a4-6f43-95a087c92367@postgrespro.ru
      fd26f782
    • Michael Paquier's avatar
      Fix progress reporting of REINDEX CONCURRENTLY · e66bcfb4
      Michael Paquier authored
      This addresses a couple of issues with the so-said subject:
      - Report the correct parent relation with the index actually being
      rebuilt or validated.  Previously, the command status remained set to
      the last index created for the progress of the index build and
      validation, which would be incorrect when working on a table that has
      more than one index.
      - Use the correct phase when waiting before the drop of the old
      indexes.  Previously, this was reported with the same status as when
      waiting before the old indexes are marked as dead.
      
      Author: Matthias van de Meent, Michael Paquier
      Discussion: https://postgr.es/m/CAEze2WhqFgcwe1_tv=sFYhLWV2AdpfukumotJ6JNcAOQs3jufg@mail.gmail.com
      Backpatch-through: 12
      e66bcfb4
    • Tom Lane's avatar
      Add for_each_from, to simplify loops starting from non-first list cells. · 56fe0089
      Tom Lane authored
      We have a dozen or so places that need to iterate over all but the
      first cell of a List.  Prior to v13 this was typically written as
      	for_each_cell(lc, lnext(list_head(list)))
      Commit 1cff1b95 changed these to
      	for_each_cell(lc, list, list_second_cell(list))
      This patch introduces a new macro for_each_from() which expresses
      the start point as a list index, allowing these to be written as
      	for_each_from(lc, list, 1)
      This is marginally more efficient, since ForEachState.i can be
      initialized directly instead of backing into it from a ListCell
      address.  It also seems clearer and less typo-prone.
      
      Some of the remaining uses of for_each_cell() look like they could
      profitably be changed to for_each_from(), but here I confined myself
      to changing uses of list_second_cell().
      
      Also, fix for_each_cell_setup() and for_both_cell_setup() to
      const-ify their arguments; that's a simple oversight in 1cff1b95.
      
      Back-patch into v13, on the grounds that (1) the const-ification
      is a minor bug fix, and (2) it's better for back-patching purposes
      if we only have two ways to write these loops rather than three.
      
      In HEAD, also remove list_third_cell() and list_fourth_cell(),
      which were also introduced in 1cff1b95, and are unused as of
      cc99baa4.  It seems unlikely that any third-party code would
      have started to use them already; anyone who has can be directed
      to list_nth_cell instead.
      
      Discussion: https://postgr.es/m/CAApHDvpo1zj9KhEpU2cCRZfSM3Q6XGdhzuAS2v79PH7WJBkYVA@mail.gmail.com
      56fe0089
    • Michael Paquier's avatar
      Revert "Change SHA2 implementation based on OpenSSL to use EVP digest routines" · fe0a1dc5
      Michael Paquier authored
      This reverts commit e21cbb4b, as the switch to EVP routines requires a
      more careful design where we would need to have at least our wrapper
      routines return a status instead of issuing an error by themselves to
      let the caller do the error handling.  The memory handling was also
      incorrect and could cause leaks in the backend if a failure happened,
      requiring most likely a callback to do the necessary cleanup as the only
      clean way to be able to allocate an EVP context requires the use of an
      allocation within OpenSSL.  The potential rework of the wrappers also
      impacts the fallback implementation when not building with OpenSSL.
      
      Originally, prairiedog has reported a compilation failure, but after
      discussion with Tom Lane this needs a better design.
      
      Discussion: https://postgr.es/m/20200928073330.GC2316@paquier.xyz
      fe0a1dc5
  2. 28 Sep, 2020 9 commits
  3. 27 Sep, 2020 1 commit
    • Tom Lane's avatar
      Move resolution of AlternativeSubPlan choices to the planner. · 41efb834
      Tom Lane authored
      When commit bd3dadda introduced AlternativeSubPlans, I had some
      ambitions towards allowing the choice of subplan to change during
      execution.  That has not happened, or even been thought about, in the
      ensuing twelve years; so it seems like a failed experiment.  So let's
      rip that out and resolve the choice of subplan at the end of planning
      (in setrefs.c) rather than during executor startup.  This has a number
      of positive benefits:
      
      * Removal of a few hundred lines of executor code, since
      AlternativeSubPlans need no longer be supported there.
      
      * Removal of executor-startup overhead (particularly, initialization
      of subplans that won't be used).
      
      * Removal of incidental costs of having a larger plan tree, such as
      tree-scanning and copying costs in the plancache; not to mention
      setrefs.c's own costs of processing the discarded subplans.
      
      * EXPLAIN no longer has to print a weird (and undocumented)
      representation of an AlternativeSubPlan choice; it sees only the
      subplan actually used.  This should mean less confusion for users.
      
      * Since setrefs.c knows which subexpression of a plan node it's
      working on at any instant, it's possible to adjust the estimated
      number of executions of the subplan based on that.  For example,
      we should usually estimate more executions of a qual expression
      than a targetlist expression.  The implementation used here is
      pretty simplistic, because we don't want to expend a lot of cycles
      on the issue; but it's better than ignoring the point entirely,
      as the executor had to.
      
      That last point might possibly result in shifting the choice
      between hashed and non-hashed EXISTS subplans in a few cases,
      but in general this patch isn't meant to change planner choices.
      Since we're doing the resolution so late, it's really impossible
      to change any plan choices outside the AlternativeSubPlan itself.
      
      Patch by me; thanks to David Rowley for review.
      
      Discussion: https://postgr.es/m/1992952.1592785225@sss.pgh.pa.us
      41efb834
  4. 26 Sep, 2020 3 commits
  5. 25 Sep, 2020 2 commits
    • Thomas Munro's avatar
      Defer flushing of SLRU files. · dee663f7
      Thomas Munro authored
      Previously, we called fsync() after writing out individual pg_xact,
      pg_multixact and pg_commit_ts pages due to cache pressure, leading to
      regular I/O stalls in user backends and recovery.  Collapse requests for
      the same file into a single system call as part of the next checkpoint,
      as we already did for relation files, using the infrastructure developed
      by commit 3eb77eba.  This can cause a significant improvement to
      recovery performance, especially when it's otherwise CPU-bound.
      
      Hoist ProcessSyncRequests() up into CheckPointGuts() to make it clearer
      that it applies to all the SLRU mini-buffer-pools as well as the main
      buffer pool.  Rearrange things so that data collected in CheckpointStats
      includes SLRU activity.
      
      Also remove the Shutdown{CLOG,CommitTS,SUBTRANS,MultiXact}() functions,
      because they were redundant after the shutdown checkpoint that
      immediately precedes them.  (I'm not sure if they were ever needed, but
      they aren't now.)
      
      Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> (parts)
      Tested-by: default avatarJakub Wartak <Jakub.Wartak@tomtom.com>
      Discussion: https://postgr.es/m/CA+hUKGLJ=84YT+NvhkEEDAuUtVHMfQ9i-N7k_o50JmQ6Rpj_OQ@mail.gmail.com
      dee663f7
    • Michael Paquier's avatar
      Remove custom memory allocation layer in pgcrypto · ca7f8e2b
      Michael Paquier authored
      PX_OWN_ALLOC was intended as a way to disable the use of palloc(), and
      over the time new palloc() or equivalent calls have been added like in
      32984d8f, making this extra layer losing its original purpose.  This
      simplifies on the way some code paths to use palloc0() rather than
      palloc() followed by memset(0).
      
      Author: Daniel Gustafsson
      Discussion: https://postgr.es/m/A5BFAA1A-B2E8-4CBC-895E-7B1B9475A527@yesql.se
      ca7f8e2b
  6. 24 Sep, 2020 7 commits
    • Tom Lane's avatar
      Fix handling of -d "connection string" in pg_dump/pg_restore. · a45bc8a4
      Tom Lane authored
      Parallel pg_dump failed if its -d parameter was a connection string
      containing any essential information other than host, port, or username.
      The same was true for pg_restore with --create.
      
      The reason is that these scenarios failed to preserve the connection
      string from the command line; the code felt free to replace that with
      just the database name when reconnecting from a pg_dump parallel worker
      or after creating the target database.  By chance, parallel pg_restore
      did not suffer this defect, as long as you didn't say --create.
      
      In practice it seems that the error would be obvious only if the
      connstring included essential, non-default SSL or GSS parameters.
      This may explain why it took us so long to notice.  (It also makes
      it very difficult to craft a regression test case illustrating the
      problem, since the test would fail in builds without those options.)
      
      Fix by refactoring so that ConnectDatabase always receives all the
      relevant options directly from the command line, rather than
      reconstructed values.  Inject a different database name, when necessary,
      by relying on libpq's rules for handling multiple "dbname" parameters.
      
      While here, let's get rid of the essentially duplicate _connectDB
      function, as well as some obsolete nearby cruft.
      
      Per bug #16604 from Zsolt Ero.  Back-patch to all supported branches.
      
      Discussion: https://postgr.es/m/16604-933f4b8791227b15@postgresql.org
      a45bc8a4
    • Robert Haas's avatar
      Fix two bugs in MaintainOldSnapshotTimeMapping. · 55b7e2f4
      Robert Haas authored
      The previous coding was confused about whether head_timestamp was
      intended to represent the timestamp for the newest bucket in the
      mapping or the oldest timestamp for the oldest bucket in the mapping.
      Decide that it's intended to be the oldest one, and repair
      accordingly.
      
      To do that, we need to do two things. First, when advancing to a
      new bucket, don't categorically set head_timestamp to the new
      timestamp. Do this only if we're blowing out the map completely
      because a lot of time has passed since we last maintained it. If
      we're replacing entries one by one, advance head_timestamp by
      1 minute for each; if we're filling in unused entries, don't
      advance head_timestamp at all.
      
      Second, fix the computation of how many buckets we need to advance.
      The previous formula would be correct if head_timestamp were the
      timestamp for the new bucket, but we're now making all the code
      agree that it's the timestamp for the oldest bucket, so adjust the
      formula accordingly.
      
      This is certainly a bug fix, but I don't feel good about
      back-patching it without the introspection tools added by commit
      aecf5ee2, and perhaps also some
      actual tests. Since back-patching the introspection tools might
      not attract sufficient support and since there are no automated
      tests of these fixes yet, I'm just committing this to master for
      now.
      
      Patch by me, reviewed by Thomas Munro, Dilip Kumar, Hamid Akhtar.
      
      Discussion: http://postgr.es/m/CA+TgmoY=aqf0zjTD+3dUWYkgMiNDegDLFjo+6ze=Wtpik+3XqA@mail.gmail.com
      55b7e2f4
    • Peter Eisentraut's avatar
      Standardize the printf format for st_size · c005eb00
      Peter Eisentraut authored
      Existing code used various inconsistent ways to printf struct stat's
      st_size member.  The type of that is off_t, which is in most cases a
      signed 64-bit integer, so use the long long int format for it.
      c005eb00
    • Robert Haas's avatar
      Add new 'old_snapshot' contrib module. · aecf5ee2
      Robert Haas authored
      You can use this to view the contents of the time to XID mapping
      which the server maintains when old_snapshot_threshold != -1.
      Being able to view that information may be interesting for users,
      and it's definitely useful for figuring out whether the mapping
      is being maintained correctly. It isn't, so that will need to be
      fixed in a subsequent commit.
      
      Patch by me, reviewed by Thomas Munro, Dilip Kumar, Hamid Akhtar.
      
      Discussion: http://postgr.es/m/CA+TgmoY=aqf0zjTD+3dUWYkgMiNDegDLFjo+6ze=Wtpik+3XqA@mail.gmail.com
      aecf5ee2
    • Robert Haas's avatar
      Expose oldSnapshotControl definition via new header. · f5ea92e8
      Robert Haas authored
      This makes it possible for code outside snapmgr.c to examine the
      contents of this data structure. This commit does not add any code
      which actually does so; a subsequent commit will make that change.
      
      Patch by me, reviewed by Thomas Munro, Dilip Kumar, Hamid Akhtar.
      
      Discussion: http://postgr.es/m/CA+TgmoY=aqf0zjTD+3dUWYkgMiNDegDLFjo+6ze=Wtpik+3XqA@mail.gmail.com
      f5ea92e8
    • Tom Lane's avatar
    • Tom Lane's avatar
      Improve behavior of tsearch_readline(), and remove t_readline(). · 83b61319
      Tom Lane authored
      Commit fbeb9da2, which added the tsearch_readline APIs, left
      t_readline() in place as a compatibility measure.  But that function
      has been unused and deprecated for twelve years now, so that seems
      like enough time to remove it.  Doing so, and merging t_readline's
      code into tsearch_readline, aids in making several useful
      improvements:
      
      * The hard-wired 4K limit on line length in tsearch data files is
      removed, by using a StringInfo buffer instead of a fixed-size buffer.
      
      * We can buy back the per-line palloc/pfree added by 3ea7e955
      in the common case where encoding conversion is not required.
      
      * We no longer need a separate pg_verify_mbstr call, as that
      functionality was folded into encoding conversion some time ago.
      
      (We could have done some of this stuff while keeping t_readline as a
      separate API, but there seems little point, since there's no reason
      for anyone to still be using t_readline directly.)
      
      Discussion: https://postgr.es/m/48A4FA71-524E-41B9-953A-FD04EF36E2E7@yesql.se
      83b61319
  7. 23 Sep, 2020 4 commits
  8. 22 Sep, 2020 5 commits
  9. 21 Sep, 2020 2 commits