Re: possible bug?

From: Takeshi Enomoto <eno_at_nyahnyahspammersnyahnyah>
Date: Thu, 14 Dec 2006 18:31:25 +0900


Thank you for your help.

> The function "floattointeger" truncates a floating point
> number to an integer.

As your example suggests, floattointeger() is not the problem.

> On a 32-bit machine you can get about
> seven digits of accuracy in a floating point number and to
> represent the sum "year*10000+month*100+31."
> precisely as an integer would require eight digits.

Initially that number would fit to float,
but I didn't pay attention to the precision.
The last digit is not accurate as you pointed out.

The correct way:

> You could make the sum double precision
> and then it would be correct in this case.

> y = year*10000+month*100+31.d
> print(y)
> print(sprintf("%15.2f",y))

Dr Takeshi Enomoto
Earth Simulator Center, JAMSTEC

ncl-talk mailing list
Received on Thu Dec 14 2006 - 02:31:25 MST

This archive was generated by hypermail 2.2.0 : Tue Dec 19 2006 - 07:55:14 MST