• Dean Rasheed's avatar
    Fix corner-case loss of precision in numeric ln(). · 43a899f4
    Dean Rasheed authored
    When deciding on the local rscale to use for the Taylor series
    expansion, ln_var() neglected to account for the fact that the result
    is subsequently multiplied by a factor of 2^(nsqrt+1), where nsqrt is
    the number of square root operations performed in the range reduction
    step, which can be as high as 22 for very large inputs. This could
    result in a loss of precision, particularly when combined with large
    rscale values, for which a large number of Taylor series terms is
    required (up to around 400).
    
    Fix by computing a few extra digits in the Taylor series, based on the
    weight of the multiplicative factor log10(2^(nsqrt+1)). It remains to
    be proven whether or not the other 8 extra digits used for the Taylor
    series is appropriate, but this at least deals with the obvious
    oversight of failing to account for the effects of the final
    multiplication.
    
    Per report from Justin AnyhowStep. Reviewed by Tom Lane.
    
    Discussion: https://postgr.es/m/16280-279f299d9c06e56f@postgresql.org
    43a899f4
numeric_big.sql 298 KB