Paul van Wamelen on Wed, 28 Aug 2002 01:12:35 -0500 (CDT) |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
Question about precision |
Dear all, In the file gen3.c there is a function called "precision" where the precision of a real number x is computed with the following code: k=2-(expo(x)>>TWOPOTBITS_IN_LONG); l=lg(x); if (l>k) k=l; return k; so essentially the number of code words used to represent x, except if the exponent is small and then essentially 2 plus the power of 2^32 that gives 10^(-exponent). But shouldn't this be 2 plus the power of 2^32 that gives 10^(-exponent+relative precision as number of decimal digits), or k=lg(x)-(expo(x)>>TWOPOTBITS_IN_LONG); ? If I am correct, why aren't all kinds of things that use the precision function broken? Thanks! Paul van Wamelen