• Tom Lane's avatar
    Increase distance between flush requests during bulk file copies. · 643c27e3
    Tom Lane authored
    copy_file() reads and writes data 64KB at a time (with default BLCKSZ),
    and historically has issued a pg_flush_data request after each write.
    This turns out to interact really badly with macOS's new APFS file
    system: a large file copy takes over 100X longer than it ought to on
    APFS, as reported by Brent Dearth.  While that's arguably a macOS bug,
    it's not clear whether Apple will do anything about it in the near
    future, and in any case experimentation suggests that issuing flushes
    a bit less often can be helpful on other platforms too.
    
    Hence, rearrange the logic in copy_file() so that flush requests are
    issued once per N writes rather than every time through the loop.
    I set the FLUSH_DISTANCE to 32MB on macOS (any less than that still
    results in a noticeable speed degradation on APFS), but 1MB elsewhere.
    In limited testing on Linux and FreeBSD, this seems slightly faster
    than the previous code, and certainly no worse.  It helps noticeably
    on macOS even with the older HFS filesystem.
    
    A simpler change would have been to just increase the size of the
    copy buffer without changing the loop logic, but that seems likely
    to trash the processor cache without really helping much.
    
    Back-patch to 9.6 where we introduced msync() as an implementation
    option for pg_flush_data().  The problem seems specific to APFS's
    mmap/msync support, so I don't think we need to go further back.
    
    Discussion: https://postgr.es/m/CADkxhTNv-j2jw2g8H57deMeAbfRgYBoLmVuXkC=YCFBXRuCOww@mail.gmail.com
    643c27e3
copydir.c 5.87 KB