• David Rowley's avatar
    Fix incorrect hash table resizing code in simplehash.h · dc23c77d
    David Rowley authored
    This fixes a bug in simplehash.h which caused an incorrect size mask to be
    used when the hash table grew to SH_MAX_SIZE (2^32).  The code was
    incorrectly setting the size mask to 0 when the hash tables reached the
    maximum possible number of buckets.  This would result always trying to
    use the 0th bucket causing an  infinite loop of trying to grow the hash
    table due to there being too many collisions.
    
    Seemingly it's not that common for simplehash tables to ever grow this big
    as this bug dates back to v10 and nobody seems to have noticed it before.
    However, probably the most likely place that people would notice it would
    be doing a large in-memory Hash Aggregate with something close to at least
    2^31 groups.
    
    After this fix, the code now works correctly with up to within 98% of 2^32
    groups and will fail with the following error when trying to insert any
    more items into the hash table:
    
    ERROR:  hash table size exceeded
    
    However, the work_mem (or hash_mem_multiplier in newer versions) settings
    will generally cause Hash Aggregates to spill to disk long before reaching
    that many groups.  The minimal test case I did took a work_mem setting of
    over 192GB to hit the bug.
    
    simplehash hash tables are used in a few other places such as Bitmap Index
    Scans, however, again the size that the hash table can become there is
    also limited to work_mem and it would take a relation of around 16TB
    (2^31) pages and a very large work_mem setting to hit this.  With smaller
    work_mem values the table would become lossy and never grow large enough
    to hit the problem.
    
    Author: Yura Sokolov
    Reviewed-by: David Rowley, Ranier Vilela
    Discussion: https://postgr.es/m/b1f7f32737c3438136f64b26f4852b96@postgrespro.ru
    Backpatch-through: 10, where simplehash.h was added
    dc23c77d
simplehash.h 32.1 KB