• Dean Rasheed's avatar
    Fix corner-case loss of precision in numeric pow() calculation · 18a02ad2
    Dean Rasheed authored
    Commit 7d9a4737 greatly improved the
    accuracy of the numeric transcendental functions, however it failed to
    consider the case where the result from pow() is close to the overflow
    threshold, for example 0.12 ^ -2345.6. For such inputs, where the
    result has more than 2000 digits before the decimal point, the decimal
    result weight estimate was being clamped to 2000, leading to a loss of
    precision in the final calculation.
    
    Fix this by replacing the clamping code with an overflow test that
    aborts the calculation early if the final result is sure to overflow,
    based on the overflow limit in exp_var(). This provides the same
    protection against integer overflow in the subsequent result scale
    computation as the original clamping code, but it also ensures that
    precision is never lost and saves compute cycles in cases that are
    sure to overflow.
    
    The new early overflow test works with the initial low-precision
    result (expected to be accurate to around 8 significant digits) and
    includes a small fuzz factor to ensure that it doesn't kick in for
    values that would not overflow exp_var(), so the overall overflow
    threshold of pow() is unchanged and consistent for all inputs with
    non-integer exponents.
    
    Author: Dean Rasheed
    Reviewed-by: Tom Lane
    Discussion: http://www.postgresql.org/message-id/CAEZATCUj3U-cQj0jjoia=qgs0SjE3auroxh8swvNKvZWUqegrg@mail.gmail.com
    See-also: http://www.postgresql.org/message-id/CAEZATCV7w+8iB=07dJ8Q0zihXQT1semcQuTeK+4_rogC_zq5Hw@mail.gmail.com
    18a02ad2
numeric_big.sql 289 KB