Paul van Wamelen on Wed, 28 Aug 2002 01:12:35 -0500 (CDT)


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Question about precision


Dear all,

In the file gen3.c there is a function called "precision" where the
precision of a real number x is computed with the following code:

    k=2-(expo(x)>>TWOPOTBITS_IN_LONG);
    l=lg(x); if (l>k) k=l;
    return k;

so essentially the number of code words used to represent x, except if the
exponent is small and then essentially 2 plus the power of 2^32 that gives
10^(-exponent). But shouldn't this be 2 plus the power of 2^32 that gives
10^(-exponent+relative precision as number of decimal digits), or
  k=lg(x)-(expo(x)>>TWOPOTBITS_IN_LONG);
?

If I am correct, why aren't all kinds of things that use the precision
function broken?

Thanks!
Paul van Wamelen