• Amit Kapila's avatar
    Add logical_decoding_work_mem to limit ReorderBuffer memory usage. · cec2edfa
    Amit Kapila authored
    Instead of deciding to serialize a transaction merely based on the
    number of changes in that xact (toplevel or subxact), this makes
    the decisions based on amount of memory consumed by the changes.
    
    The memory limit is defined by a new logical_decoding_work_mem GUC,
    so for example we can do this
    
        SET logical_decoding_work_mem = '128kB'
    
    to reduce the memory usage of walsenders or set the higher value to
    reduce disk writes. The minimum value is 64kB.
    
    When adding a change to a transaction, we account for the size in
    two places. Firstly, in the ReorderBuffer, which is then used to
    decide if we reached the total memory limit. And secondly in the
    transaction the change belongs to, so that we can pick the largest
    transaction to evict (and serialize to disk).
    
    We still use max_changes_in_memory when loading changes serialized
    to disk. The trouble is we can't use the memory limit directly as
    there might be multiple subxact serialized, we need to read all of
    them but we don't know how many are there (and which subxact to
    read first).
    
    We do not serialize the ReorderBufferTXN entries, so if there is a
    transaction with many subxacts, most memory may be in this type of
    objects. Those records are not included in the memory accounting.
    
    We also do not account for INTERNAL_TUPLECID changes, which are
    kept in a separate list and not evicted from memory. Transactions
    with many CTID changes may consume significant amounts of memory,
    but we can't really do much about that.
    
    The current eviction algorithm is very simple - the transaction is
    picked merely by size, while it might be useful to also consider age
    (LSN) of the changes for example. With the new Generational memory
    allocator, evicting the oldest changes would make it more likely
    the memory gets actually pfreed.
    
    The logical_decoding_work_mem can be set in postgresql.conf, in which
    case it serves as the default for all publishers on that instance.
    
    Author: Tomas Vondra, with changes by Dilip Kumar and Amit Kapila
    Reviewed-by: Dilip Kumar and Amit Kapila
    Tested-By: Vignesh C
    Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
    cec2edfa
guc.c 304 KB