• Tom Lane's avatar
    Remove fixed limit on the number of concurrent AllocateFile() requests. · 007556bf
    Tom Lane authored
    AllocateFile(), AllocateDir(), and some sister routines share a small array
    for remembering requests, so that the files can be closed on transaction
    failure.  Previously that array had a fixed size, MAX_ALLOCATED_DESCS (32).
    While historically that had seemed sufficient, Steve Toutant pointed out
    that this meant you couldn't scan more than 32 file_fdw foreign tables in
    one query, because file_fdw depends on the COPY code which uses
    AllocateFile().  There are probably other cases, or will be in the future,
    where this nonconfigurable limit impedes users.
    
    We can't completely remove any such limit, at least not without a lot of
    work, since each such request requires a kernel file descriptor and most
    platforms limit the number we can have.  (In principle we could
    "virtualize" these descriptors, as fd.c already does for the main VFD pool,
    but not without an additional layer of overhead and a lot of notational
    impact on the calling code.)  But we can at least let the array size be
    configurable.  Hence, change the code to allow up to max_safe_fds/2
    allocated file requests.  On modern platforms this should allow several
    hundred concurrent file_fdw scans, or more if one increases the value of
    max_files_per_process.  To go much further than that, we'd need to do some
    more work on the data structure, since the current code for closing
    requests has potentially O(N^2) runtime; but it should still be all right
    for request counts in this range.
    
    Back-patch to 9.1 where contrib/file_fdw was introduced.
    007556bf
fd.c 59.9 KB