• Dean Rasheed's avatar
    Fix corner-case errors and loss of precision in numeric_power(). · 0d6b8749
    Dean Rasheed authored
    This fixes a couple of related problems that arise when raising
    numbers to very large powers.
    
    Firstly, when raising a negative number to a very large integer power,
    the result should be well-defined, but the previous code would only
    cope if the exponent was small enough to go through power_var_int().
    Otherwise it would throw an internal error, attempting to take the
    logarithm of a negative number. Fix this by adding suitable handling
    to the general case in power_var() to cope with negative bases,
    checking for integer powers there.
    
    Next, when raising a (positive or negative) number whose absolute
    value is slightly less than 1 to a very large power, the result should
    approach zero as the power is increased. However, in some cases, for
    sufficiently large powers, this would lose all precision and return 1
    instead of 0. This was due to the way that the local_rscale was being
    calculated for the final full-precision calculation:
    
      local_rscale = rscale + (int) val - ln_dweight + 8
    
    The first two terms on the right hand side are meant to give the
    number of significant digits required in the result ("val" being the
    estimated result weight). However, this failed to account for the fact
    that rscale is clipped to a maximum of NUMERIC_MAX_DISPLAY_SCALE
    (1000), and the result weight might be less then -1000, causing their
    sum to be negative, leading to a loss of precision. Fix this by
    forcing the number of significant digits calculated to be nonnegative.
    It's OK for it to be zero (when the result weight is less than -1000),
    since the local_rscale value then includes a few extra digits to
    ensure an accurate result.
    
    Finally, add additional underflow checks to exp_var() and power_var(),
    so that they consistently return zero for cases like this where the
    result is indistinguishable from zero. Some paths through this code
    already returned zero in such cases, but others were throwing overflow
    errors.
    
    Dean Rasheed, reviewed by Yugo Nagata.
    
    Discussion: http://postgr.es/m/CAEZATCW6Dvq7+3wN3tt5jLj-FyOcUgT5xNoOqce5=6Su0bCR0w@mail.gmail.com
    0d6b8749
numeric.out 103 KB