1. 08 Aug, 2017 3 commits
    • Alvaro Herrera's avatar
      Fix replication origin-related race conditions · b2c95a37
      Alvaro Herrera authored
      Similar to what was fixed in commit 9915de6c for replication slots,
      but this time it's related to replication origins: DROP SUBSCRIPTION
      attempts to drop the replication origin, but that fails if the
      replication worker process hasn't yet marked it unused.  This causes
      failures in the buildfarm:
      ERROR:  could not drop replication origin with OID 1, in use by PID 34069
      
      Like the aforementioned commit, fix by having the process running DROP
      SUBSCRIPTION sleep until the worker marks the the replication origin
      struct as free.  This uses a condition variable on each replication
      origin shmem state struct, so that the session trying to drop can sleep
      and expect to be awakened by the process keeping the origin open.
      
      Also fix a SGML markup in the previous commit.
      
      Discussion: https://postgr.es/m/20170808001433.rozlseaf4m2wkw3n@alvherre.pgsql
      b2c95a37
    • Alvaro Herrera's avatar
      Fix inadequacies in recently added wait events · 030273b7
      Alvaro Herrera authored
      In commit 9915de6c, we introduced a new wait point for replication
      slots and incorrectly labelled it as wait event PG_WAIT_LOCK.  That's
      wrong, so invent an appropriate new wait event instead, and document it
      properly.
      
      While at it, fix numerous other problems in the vicinity:
      - two different walreceiver wait events were being mixed up in a single
        wait event (which wasn't documented either); split it out so that they
        can be distinguished, and document the new events properly.
      
      - ParallelBitmapPopulate was documented but didn't exist.
      
      - ParallelBitmapScan was not documented (I think this should be called
        "ParallelBitmapScanInit" instead.)
      
      - Logical replication wait events weren't documented
      
      - various symbols had been added in dartboard order in various places.
        Put them in alphabetical order instead, as was originally intended.
      
      Discussion: https://postgr.es/m/20170808181131.mu4fjepuh5m75cyq@alvherre.pgsql
      030273b7
    • Noah Misch's avatar
      Disclaim xmltable() support for non-UTF8 databases. · b4a2eea0
      Noah Misch authored
      The xmltable() implementation mirrors xpath(), including its lack of
      character encoding awareness.
      b4a2eea0
  2. 07 Aug, 2017 14 commits
    • Tom Lane's avatar
      Stamp 10beta3. · 8d644237
      Tom Lane authored
      8d644237
    • Tom Lane's avatar
      Skip test for IPC::Run if user is overriding our search for PROVE. · 8014d2af
      Tom Lane authored
      The check for IPC::Run we added in commit c254970a is useful in simple
      cases, but there are real use-cases where "prove" is coming from a
      different Perl installation than the "perl" we want to use to build.
      In such cases asking whether "perl" knows about IPC::Run is irrelevant
      and can cause an unnecessary configure failure.  Hence, if user has
      specified a value for PROVE, skip the IPC::Run check.  Per discussion
      with Andrew Dunstan.
      
      Discussion: https://postgr.es/m/E1dcE5n-0005Sk-UE@gemulon.postgresql.org
      8014d2af
    • Peter Eisentraut's avatar
      Update SQL features list · cdc47d1f
      Peter Eisentraut authored
      cdc47d1f
    • Peter Eisentraut's avatar
      Translation updates · f7668b2b
      Peter Eisentraut authored
      Source-Git-URL: git://git.postgresql.org/git/pgtranslation/messages.git
      Source-Git-Hash: 1a0b5e655d7871506c2b1c7ba562c2de6b6a55de
      f7668b2b
    • Tom Lane's avatar
      Last-minute updates for release notes. · a8b37ebe
      Tom Lane authored
      Security: CVE-2017-7546, CVE-2017-7547, CVE-2017-7548
      a8b37ebe
    • Peter Eisentraut's avatar
      Fix local/remote attribute mix-up in logical replication · fca17a93
      Peter Eisentraut authored
      This would lead to failures if local and remote tables have a different
      column order.  The tests previously didn't catch that because they only
      tested the initial data copy.  So add another test that exercises the
      apply worker.
      
      Author: Petr Jelinek <petr.jelinek@2ndquadrant.com>
      fca17a93
    • Peter Eisentraut's avatar
      Fix handling of dropped columns in logical replication · 0e58455d
      Peter Eisentraut authored
      The relation attribute map was not initialized for dropped columns,
      leading to errors later on.
      
      Author: Petr Jelinek <petr.jelinek@2ndquadrant.com>
      Reported-by: default avatarScott Milliken <scott@deltaex.com>
      Bug: #14769
      0e58455d
    • Tom Lane's avatar
      Require update permission for the large object written by lo_put(). · 8d988191
      Tom Lane authored
      lo_put() surely should require UPDATE permission, the same as lowrite(),
      but it failed to check for that, as reported by Chapman Flack.  Oversight
      in commit c50b7c09; backpatch to 9.4 where that was introduced.
      
      Tom Lane and Michael Paquier
      
      Security: CVE-2017-7548
      8d988191
    • Noah Misch's avatar
      Again match pg_user_mappings to information_schema.user_mapping_options. · e568e1ee
      Noah Misch authored
      Commit 3eefc510 claimed to make
      pg_user_mappings enforce the qualifications user_mapping_options had
      been enforcing, but its removal of a longstanding restriction left them
      distinct when the current user is the subject of a mapping yet has no
      server privileges.  user_mapping_options emits no rows for such a
      mapping, but pg_user_mappings includes full umoptions.  Change
      pg_user_mappings to show null for umoptions.  Back-patch to 9.2, like
      the above commit.
      
      Reviewed by Tom Lane.  Reported by Jeff Janes.
      
      Security: CVE-2017-7547
      e568e1ee
    • Heikki Linnakangas's avatar
      Don't allow logging in with empty password. · bf6b9e94
      Heikki Linnakangas authored
      Some authentication methods allowed it, others did not. In the client-side,
      libpq does not even try to authenticate with an empty password, which makes
      using empty passwords hazardous: an administrator might think that an
      account with an empty password cannot be used to log in, because psql
      doesn't allow it, and not realize that a different client would in fact
      allow it. To clear that confusion and to be be consistent, disallow empty
      passwords in all authentication methods.
      
      All the authentication methods that used plaintext authentication over the
      wire, except for BSD authentication, already checked that the password
      received from the user was not empty. To avoid forgetting it in the future
      again, move the check to the recv_password_packet function. That only
      forbids using an empty password with plaintext authentication, however.
      MD5 and SCRAM need a different fix:
      
      * In stable branches, check that the MD5 hash stored for the user does not
      not correspond to an empty string. This adds some overhead to MD5
      authentication, because the server needs to compute an extra MD5 hash, but
      it is not noticeable in practice.
      
      * In HEAD, modify CREATE and ALTER ROLE to clear the password if an empty
      string, or a password hash that corresponds to an empty string, is
      specified. The user-visible behavior is the same as in the stable branches,
      the user cannot log in, but it seems better to stop the empty password from
      entering the system in the first place. Secondly, it is fairly expensive to
      check that a SCRAM hash doesn't correspond to an empty string, because
      computing a SCRAM hash is much more expensive than an MD5 hash by design,
      so better avoid doing that on every authentication.
      
      We could clear the password on CREATE/ALTER ROLE also in stable branches,
      but we would still need to check at authentication time, because even if we
      prevent empty passwords from being stored in pg_authid, there might be
      existing ones there already.
      
      Reported by Jeroen van der Ham, Ben de Graaff and Jelte Fennema.
      
      Security: CVE-2017-7546
      bf6b9e94
    • Peter Eisentraut's avatar
      Fix function name in code comment · 86524f03
      Peter Eisentraut authored
      Reported-by: default avatarPeter Geoghegan <pg@bowt.ie>
      86524f03
    • Peter Eisentraut's avatar
    • Peter Eisentraut's avatar
      Downgrade subscription refresh messages to DEBUG1 · 6f81306e
      Peter Eisentraut authored
      The NOTICE messages about tables being added or removed during
      subscription refresh would be incorrect and possibly confusing if the
      transaction rolls back, so silence them but keep them available for
      debugging.
      
      Discussion: https://www.postgresql.org/message-id/CAD21AoAvaXizc2h7aiNyK_i0FQSa-tmhpdOGwbhh7Jy544Ad4Q%40mail.gmail.com
      6f81306e
    • Tom Lane's avatar
      Update RELEASE_CHANGES' example of branch name format. · 655727d9
      Tom Lane authored
      We're planning to put an underscore before the major version number in
      branch names for v10 and later.  Make sure the recipe in RELEASE_CHANGES
      reflects that.
      
      In passing, add a reminder to consider doing pgindent right before
      the branch.
      
      Discussion: https://postgr.es/m/E1dAkjZ-0003MG-0U@gemulon.postgresql.org
      655727d9
  3. 06 Aug, 2017 2 commits
  4. 05 Aug, 2017 8 commits
  5. 04 Aug, 2017 7 commits
    • Robert Haas's avatar
      hash: Immediately after a bucket split, try to clean the old bucket. · ff98a5e1
      Robert Haas authored
      If it works, then we won't be storing two copies of all the tuples
      that were just moved.  If not, VACUUM will still take care of it
      eventually.  Per a report from AP and analysis from Amit Kapila, it
      seems that a bulk load can cause splits fast enough that VACUUM won't
      deal with the problem in time to prevent bloat.
      
      Amit Kapila; I rewrote the comment.
      
      Discussion: http://postgr.es/m/20170704105728.mwb72jebfmok2nm2@zip.com.au
      ff98a5e1
    • Tom Lane's avatar
      First-draft release notes for 9.6.4. · 03378c4d
      Tom Lane authored
      As usual, the release notes for other branches will be made by cutting
      these down, but put them up for community review first.
      03378c4d
    • Peter Eisentraut's avatar
      Message style improvements · 26d40ada
      Peter Eisentraut authored
      26d40ada
    • Robert Haas's avatar
      hash: Increase the number of possible overflow bitmaps by 8x. · 620b49a1
      Robert Haas authored
      Per a report from AP, it's not that hard to exhaust the supply of
      bitmap pages if you create a table with a hash index and then insert a
      few billion rows - and then you start getting errors when you try to
      insert additional rows.  In the particular case reported by AP,
      there's another fix that we can make to improve recycling of overflow
      pages, which is another way to avoid the error, but there may be other
      cases where this problem happens and that fix won't help.  So let's
      buy ourselves as much headroom as we can without rearchitecting
      anything.
      
      The comments claim that the old limit was 64GB, but it was really
      only 32GB, because we didn't use all the bits in the page for bitmap
      bits - only the largest power of 2 that could fit after deducting
      space for the page header and so forth.  Thus, we have 4kB per page
      for bitmap bits, not 8kB.  The new limit is thus actually 8 times the
      old *real* limit but only 4 times the old *purported* limit.
      
      Since this breaks on-disk compatibility, bump HASH_VERSION.  We've
      already done this earlier in this release cycle, so this doesn't cause
      any incremental inconvenience for people using pg_upgrade from
      releases prior to v10.  However, users who use pg_upgrade to reach
      10beta3 or later from 10beta2 or earlier will need to REINDEX any hash
      indexes again.
      
      Amit Kapila and Robert Haas
      
      Discussion: http://postgr.es/m/20170704105728.mwb72jebfmok2nm2@zip.com.au
      620b49a1
    • Tom Lane's avatar
      Apply ALTER ... SET NOT NULL recursively in ALTER ... ADD PRIMARY KEY. · c30f1770
      Tom Lane authored
      If you do ALTER COLUMN SET NOT NULL against an inheritance parent table,
      it will recurse to mark all the child columns as NOT NULL as well.  This
      is necessary for consistency: if the column is labeled NOT NULL then
      reading it should never produce nulls.
      
      However, that didn't happen in the case where ALTER ... ADD PRIMARY KEY
      marks a target column NOT NULL that wasn't before.  That was questionable
      from the beginning, and now Tushar Ahuja points out that it can lead to
      dump/restore failures in some cases.  So let's make that case recurse too.
      
      Although this is meant to fix a bug, it's enough of a behavioral change
      that I'm pretty hesitant to back-patch, especially in view of the lack
      of similar field complaints.  It doesn't seem to be too late to put it
      into v10 though.
      
      Michael Paquier, editorialized on slightly by me
      
      Discussion: https://postgr.es/m/b8794d6a-38f0-9d7c-ad4b-e85adf860fc9@enterprisedb.com
      c30f1770
    • Tom Lane's avatar
      Disallow SSL session tickets. · 97d3a0b0
      Tom Lane authored
      We don't actually support session tickets, since we do not create an SSL
      session identifier.  But it seems that OpenSSL will issue a session ticket
      on-demand anyway, which will then fail when used.  This results in
      reconnection failures when using ticket-aware client-side SSL libraries
      (such as the Npgsql .NET driver), as reported by Shay Rojansky.
      
      To fix, just tell OpenSSL not to issue tickets.  At some point in the
      far future, we might consider enabling tickets instead.  But the security
      implications of that aren't entirely clear; and besides it would have
      little benefit except for very short-lived database connections, which is
      Something We're Bad At anyhow.  It would take a lot of other work to get
      to a point where that would really be an exciting thing to do.
      
      While at it, also tell OpenSSL not to use a session cache.  This doesn't
      really do anything, since a backend would never populate the cache anyway,
      but it might gain some micro-efficiencies and/or reduce security
      exposures.
      
      Patch by me, per discussion with Heikki Linnakangas and Shay Rojansky.
      Back-patch to all supported versions.
      
      Discussion: https://postgr.es/m/CADT4RqBU8N-csyZuzaook-c795dt22Zcwg1aHWB6tfVdAkodZA@mail.gmail.com
      97d3a0b0
    • Peter Eisentraut's avatar
      Further unify ROLE and USER command grammar rules · b3744812
      Peter Eisentraut authored
      ALTER USER ... SET did not support all the syntax variants of ALTER ROLE
      ...  SET.  Fix that, and to avoid further deviations of this kind, unify
      many the grammar rules for ROLE/USER/GROUP commands.
      Reported-by: default avatarPavel Golub <pavel@microolap.com>
      b3744812
  6. 03 Aug, 2017 6 commits
    • Tom Lane's avatar
      Fix pg_dump/pg_restore to emit REFRESH MATERIALIZED VIEW commands last. · 3eb9a5e7
      Tom Lane authored
      Because we push all ACL (i.e. GRANT/REVOKE) restore steps to the end,
      materialized view refreshes were occurring while the permissions on
      referenced objects were still at defaults.  This led to failures if,
      say, an MV owned by user A reads from a table owned by user B, even
      if B had granted the necessary privileges to A.  We've had multiple
      complaints about that type of restore failure, most recently from
      Jordan Gigov.
      
      The ideal fix for this would be to start treating ACLs as dependency-
      sortable objects, rather than hard-wiring anything about their dump order
      (the existing approach is a messy kluge dating to commit dc0e76ca).
      But that's going to be a rather major change, and it certainly wouldn't
      lead to a back-patchable fix.  As a short-term solution, convert the
      existing two-pass hack (ie, normal objects then ACLs) to a three-pass hack,
      ie, normal objects then ACLs then matview refreshes.  Because this happens
      in RestoreArchive(), it will also fix the problem when restoring from an
      existing archive-format dump.
      
      (Note this means that if a matview refresh would have failed under the
      permissions prevailing at dump time, it'll fail during restore as well.
      We'll define that as user error rather than something we should try
      to work around.)
      
      To avoid performance loss in parallel restore, we need the matview
      refreshes to still be parallelizable.  Hence, clean things up enough
      so that both ACLs and matviews are handled by the parallel restore
      infrastructure, instead of reverting back to serial restore for ACLs.
      There is still a final serial step, but it shouldn't normally have to
      do anything; it's only there to try to recover if we get stuck due to
      some problem like unresolved circular dependencies.
      
      Patch by me, but it owes something to an earlier attempt by Kevin Grittner.
      Back-patch to 9.3 where materialized views were introduced.
      
      Discussion: https://postgr.es/m/28572.1500912583@sss.pgh.pa.us
      3eb9a5e7
    • Alvaro Herrera's avatar
      Fix build on zlib-less environments · 9a3b5d3a
      Alvaro Herrera authored
      Commit 4d57e838 added support for getting I/O errors out of zlib,
      but it introduced a portability problem for systems without zlib.
      Repair by wrapping the zlib call inside #ifdef and restore the original
      code in the other branch.
      
      This serves to illustrate the inadequacy of the zlib abstraction in
      pg_backup_archiver: there is no way to call gzerror() in that
      abstraction.  This means that the several places that call GZREAD and
      GZWRITE are currently doing error reporting wrongly, but ENOTIME to get
      it fixed before next week's release set.
      
      Backpatch to 9.4, like the commit that introduced the problem.
      9a3b5d3a
    • Robert Haas's avatar
    • Robert Haas's avatar
    • Robert Haas's avatar
      Allow a foreign table CHECK constraint to be initially NOT VALID. · 86705aa8
      Robert Haas authored
      For a table, the constraint can be considered validated immediately,
      because the table must be empty.  But for a foreign table this is
      not necessarily the case.
      
      Fixes a bug in commit f27a6b15.
      
      Amit Langote, with some changes by me.
      
      Discussion: http://postgr.es/m/d2b7419f-4a71-cf86-cc99-bfd0f359a1ea@lab.ntt.co.jp
      86705aa8
    • Robert Haas's avatar
      Improve ExecModifyTable comments. · 12a34f59
      Robert Haas authored
      Some of these comments wrongly implied that only an AFTER ROW trigger
      will cause a 'wholerow' attribute to be present for a foreign table,
      but a BEFORE ROW trigger can have the same effect.  Others implied
      that it would always be present for a foreign table, but that's not
      true either.
      
      Etsuro Fujita and Robert Haas
      
      Discussion: http://postgr.es/m/10026bc7-1403-ef85-9e43-c6100c1cc0e3@lab.ntt.co.jp
      12a34f59