Where appropriate

Report by Nicolas Barbier, Tatsuo Ishii, MauMau

America/Metlakatla hasn't been in the IANA database all that long, so
some installations might not have it.  It does seem worthwhile to test
with a fractional-minute GMT offset, but we can get that from almost
any pre-1900 date; I chose Europe/Paris, whose LMT offset from Greenwich
should be pretty darn well established.

Also, assuming that Mars/Mons_Olympus will never be in the IANA database
seems less than future-proof, so let's use a more fanciful location for
the bad-zone-name check.

Per complaint from Christoph Berg.

config.pl and buildenv.pl can be used to customize build settings when
using MSVC.  They should never get committed into the common source tree.

Back-patch to 9.0; it looks like the rules were different in 8.4.

Michael Paquier

Amit Langote

The leak is fairly small and rare, but a leak nevertheless.

Per Coverity report. Backpatch to 9.2, where pg_receivexlog was added.
pg_basebackup shares the code, but it always exits on error, so there is
no real leak.

The original coding for ALTER SYSTEM made a fundamentally bogus assumption
that postgresql.auto.conf could be sought relative to the main config file
if we hadn't yet determined the value of data_directory.  This fails for
common arrangements with the config file elsewhere, as reported by
Christoph Berg.

The simplest fix is to not try to read postgresql.auto.conf until after
SelectConfigFiles has chosen (and locked down) the data_directory setting.

Because of the logic in ProcessConfigFile for handling resetting of GUCs
that've been removed from the config file, we cannot easily read the main
and auto config files separately; so this patch adopts a brute force
approach of reading the main config file twice during postmaster startup.
That's a tad ugly, but the actual time cost is likely to be negligible,
and there's no time for a more invasive redesign before beta.

With this patch, any attempt to set data_directory via ALTER SYSTEM
will be silently ignored.  It would probably be better to throw an
error, but that can be dealt with later.  This bug, however, would
prevent any testing of ALTER SYSTEM by a significant fraction of the
userbase, so it seems important to get it fixed before beta.

There's no longer much pressure to switch the default GIN opclass for
jsonb, but there was still some unhappiness with the name "jsonb_hash_ops",
since hashing is no longer a distinguishing property of that opclass,
and anyway it seems like a relatively minor detail.  At the suggestion of
Heikki Linnakangas, we'll use "jsonb_path_ops" instead; that captures the
important characteristic that each index entry depends on the entire path
from the document root to the indexed value.

Also add a user-facing explanation of the implementation properties of
these two opclasses.

Per discussion, this seems like a more consistent choice of name.

Fabrízio de Royes Mello, after a suggestion by Peter Eisentraut;
some additional documentation wordsmithing by me

Document existence operator adequately; fix obsolete claim that no
Unicode-escape semantic checks happen on input (it's still true for
json, but not for jsonb); improve examples; assorted wordsmithing.

When returning rows from a bitmap, as done with partial match queries, we
would get stuck in an infinite loop if the bitmap contained a lossy page
reference.

This bug is new in master, it was introduced by the patch to allow skipping
items refuted by other entries in GIN scans.

Report and fix by Alexander Korotkov

Usually the search would find plain "tclsh" without any trouble,
but some installations might only have the version-numbered flavor
of that program.

No compatibility problems have been reported with 8.6, so we might
as well back-patch this to all active branches.

Christoph Berg

It takes two arguments, not one.

reserveFromBuffer() failed to consider the possibility that it needs to
more-than-double the current buffer size.  Beyond that, it seems likely
that we'd someday need to worry about integer overflow of the buffer
length variable.  Rather than reinvent the logic that's already been
debugged in stringinfo.c, let's go back to using that logic.  We can
still have the same targeted API, but we'll rely on stringinfo.c to
manage reallocation.

Per report from Alexander Korotkov.

I started out with the intention of just fixing the info about the jsonb
operator classes, but soon found myself copy-editing most of the JSON
material.  Hopefully it's more readable now.

These functions were relying on typcategory to identify arrays and
composites, which is not reliable and not the normal way to do it.
Using typcategory to identify boolean, numeric types, and json itself is
also pretty questionable, though the code in those cases didn't seem to be
at risk of anything worse than wrong output.  Instead, use the standard
lsyscache functions to identify arrays and composites, and rely on a direct
check of the type OID for the other cases.

In HEAD, also be sure to look through domains so that a domain is treated
the same as its base type for conversions to JSON.  However, this is a
small behavioral change; given the lack of field complaints, we won't
back-patch it.

In passing, refactor so that there's only one copy of the code that decides
which conversion strategy to apply, not multiple copies that could (and
have) gotten out of sync.

Post-commit review identified a number of places where addition was
used instead of multiplication or memory wasn't zeroed where it should
have been.  This commit also fixes one case where a structure member
was mis-initialized, and moves another memory allocation closer to
the place where the allocated storage is used for clarity.

Andres Freund

It's legal to configure wal_level=logical and max_replication_slots=0
simultaneously.

Andres Freund

This code really needs to be refactored so that there aren't so many copies
that can diverge.  Not to mention that this whole approach is probably
wrong.  But for the moment I'll just stick my finger in the dike.
Per report from Michael Paquier.

Dunno who had the cute idea of labeling jsonb as typcategory 'C',
but it is not a composite type.  Label it 'U', since that's what
json is using.

Fix JSONB_MAX_ELEMS and JSONB_MAX_PAIRS macros to use CB_MASK in the
calculation. JENTRY_POSMASK happens to have the same value at the moment,
but that's just coincidental.

Refactor jsonb iterator functions, for readability.

Get rid of the JENTRY_ISFIRST flag. Whenever we handle JEntrys, we have
access to the whole array and have enough context information to know
which entry is the first. This frees up one bit in the JEntry header for
future use. While we're at it, shuffle the JEntry bits so that boolean
true and false go together, for aesthetic reasons.

Bump catalog version as this changes the on-disk format slightly.

Change the key representation so that values that would exceed 127 bytes
are hashed into short strings, and so that the original JSON datatype of
each value is recorded in the index.  The hashing rule eliminates the major
objection to having this opclass be the default for jsonb, namely that it
could fail for plausible input data (due to GIN's restrictions on maximum
key length).  Preserving datatype information doesn't really buy us much
right now, but it requires no extra space compared to the previous way,
and it might be useful later.

Also, change the consistency-checking functions to request recheck for
exists (jsonb ? text) and related operators.  The original analysis that
this is an exactly checkable query was incorrect, since the index does
not preserve information about whether a key appears at top level in
the indexed JSON object.  Add a test case demonstrating the problem.

Make some other, mostly cosmetic improvements to the code in jsonb_gin.c
as well.

catversion bump due to on-disk data format change in jsonb_ops indexes.

Move the functions around to group related functions together. Remove
binequal argument from lengthCompareJsonbStringValue, moving that
responsibility to lengthCompareJsonbPair. Fix typo in comment.

This speeds up text to jsonb parsing and hstore to jsonb conversions
somewhat.

Ensure that ecpg preprocessor output files are rebuilt when re-testing
after a change in the ecpg preprocessor itself, or a change in any of
several include files that get copied verbatim into the output files.
The lack of these dependencies was what created problems for Kevin Grittner
after the recent pgindent run.  There's no way for --enable-depend to
discover these dependencies automatically, so we've gotta put them into
the Makefiles by hand.

While at it, reduce the amount of duplication in the ecpg invocations.

Back in 8.3, we installed permissions checks in these functions (see
commits 8bc225e7 and cc26599b).  But we forgot to document that
anywhere in the user-facing docs; it did get mentioned in the 8.3 release
notes, but nobody's looking at that any more.  Per gripe from Suya Huang.

Per discussion, the old value of 128MB is ridiculously small on modern
machines; in fact, it's not even any larger than the default value of
shared_buffers, which it certainly should be.  Increase to 4GB, which
is unlikely to be any worse than the old default for anyone, and should
be noticeably better for most.  Eventually we might have an autotuning
scheme for this setting, but the recent attempt crashed and burned,
so for now just do this.

This reverts commit ee1e5662, as well as
a remarkably large number of followup commits, which were mostly concerned
with the fact that the implementation didn't work terribly well.  It still
doesn't: we probably need some rather basic work in the GUC infrastructure
if we want to fully support GUCs whose default varies depending on the
value of another GUC.  Meanwhile, it also emerged that there wasn't really
consensus in favor of the definition the patch tried to implement (ie,
effective_cache_size should default to 4 times shared_buffers).  So whack
it all back to where it was.  In a followup commit, I'll do what was
recently agreed to, which is to simply change the default to a higher
value.

Commit 4318daec broke it.  The change in
sub-second precision at extreme dates is normal.  The inconsistent
truncation vs. rounding is essentially a bug, albeit a longstanding one.
Back-patch to 8.4, like the causative commit.

Previous commit was confused about the case we're handling: actually,
what the patch is dealing with is platforms that have optreset, *and*
have <getopt.h>, but the latter fails to declare the former.  Because
we use a linking probe to set HAVE_INT_OPTRESET, we need to be sure we
have a declaration even if <getopt.h> doesn't think it exists.

Reportedly, some versions of mingw are like that, and it seems plausible
in general that older platforms might be that way.  However, we'd
determined experimentally that just doing "extern int" conflicts with
the way Cygwin declares these variables, so explicitly exclude Cygwin.

Michael Paquier, tweaked by me to hopefully not break Cygwin

To-be-deleted list pages contain no useful information, as they are being
deleted, but we must still protect the writes from being torn by a crash
after a partial write. To do that, re-initialize the pages on WAL replay.

Jeff Janes caught this with a test program to test partial writes.
Backpatch to all supported versions.

Michael Paquier

If the server sends a long stream of data, and the server + network are
consistently fast enough to force the recv() loop in pqReadData() to
iterate until libpq's input buffer is full, then upon processing the last
incomplete message in each bufferload we'd usually double the buffer size,
due to supposing that we didn't have enough room in the buffer to finish
collecting that message.  After filling the newly-enlarged buffer, the
cycle repeats, eventually resulting in an out-of-memory situation (which
would be reported misleadingly as "lost synchronization with server").
Of course, we should not enlarge the buffer unless we still need room
after discarding already-processed messages.

This bug dates back quite a long time: pqParseInput3 has had the behavior
since perhaps 2003, getCopyDataMessage at least since commit 70066eb1
in 2008.  Probably the reason it's not been isolated before is that in
common environments the recv() loop would always be faster than the server
(if on the same machine) or faster than the network (if not); or at least
it wouldn't be slower consistently enough to let the buffer ramp up to a
problematic size.  The reported cases involve Windows, which perhaps has
different timing behavior than other platforms.

Per bug #7914 from Shin-ichi Morita, though this is different from his
proposed solution.  Back-patch to all supported branches.

The previous behavior was to restart immediately, which was generally
viewed as less useful.

Petr Jelinek, with some adjustments by me.

Oops, I didn't realize that contrib/hstore refers to jsonb stuff.