Re: A truncation issue with tointeger()?

From: Dennis Shea <shea_at_nyahnyahspammersnyahnyah>
Date: Thu Jan 06 2011 - 16:18:39 MST

Rounding issue.

  tmpV = (/ -1.05, 15.75 /)
  dvar = 0.15
  tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
  print(tmp1) ; 112
  print(sprintf("%5.2f", tmp1)) ; 112.00
  print(sprintf("%12.5f", tmp1)) ; 111.99999

   i1 = toint(tmp1) ; 111
   i2 = floattoint(tmp1) ; 111
   i3 = round(tmp,3) ; 112

On 01/06/2011 04:06 PM, Emilie Vanvyve wrote:
> No problem :-). The truncation doesn't bother me as such, it is why it
> occurs in this particular case that does.
>
> The page you refer to says that /"If the input variable is float or
> double, then any fractional part of the input values is truncated. The
> dimensionality of the output matches that of the input."/.
>
> So, if my number is 112, there is no fractional part and there should
> not be any truncation and tointeger should return 112 (which it
> doesn't). If my number is 111.9999999, then the 0.999999 will be
> truncated and tointeger should return 111.
>
> Emilie
>
> On Jan 6, 2011, at 15:55, James Correia wrote:
>
>> I can replicate what you did. And then I did a:
>> print(numeric2int(g,1)) and the answer was 112 (no decimal)
>> print(numeric2int(g,0)) and the answer was 111 (no decimal)
>>
>> In the function description of numeric2int option 1 is rounded, option
>> 0 is truncated. so perhaps tointeger is truncated.
>>
>> http://www.ncl.ucar.edu/Document/Functions/Built-in/tointeger.shtml
>>
>> Indeed it says in the documentation that it is truncated.
>>
>> Pardon my guessing in the previous email
>> jimmyc
>>
>> On Thu, Jan 6, 2011 at 4:46 PM, Emilie Vanvyve <evanvyve@ucar.edu
>> <mailto:evanvyve@ucar.edu>> wrote:
>>
>> Hi James,
>> I am not sure I understand what you mean. My 112 value is from a
>> float variable. I would have expected it to display as
>> 11.99999999999999 or something if the calculation wasn't leading
>> to an exact 112 (which it should, but who knows) and I would then
>> understand why tointeger() gives me 111 (the .99999999 part being
>> then chopped).
>>
>> Emilie
>>
>> On Jan 6, 2011, at 15:43, James Correia wrote:
>>
>>> I don't know if this is related or not, but numeric types dont
>>> have the decimal, while float and integer do. It looks like
>>> tointeger must do rounding, but I am just guessing.
>>> jimmyc
>>>
>>> On Thu, Jan 6, 2011 at 4:27 PM, Emilie Vanvyve <evanvyve@ucar.edu
>>> <mailto:evanvyve@ucar.edu>> wrote:
>>>
>>> Hello,
>>>
>>> I just bumped into the below problem in my script and I
>>> cannot figure out why it happens. I was hoping someone would
>>> have an idea...
>>>
>>> I have the following 2 float variables:
>>> tmpV = (/ -1.05, 15.75 /) ; lower and upper bounds of a new
>>> array to be later defined with a dvar increment
>>> dvar = 0.15
>>>
>>> I calculate tmp1 as:
>>> tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
>>> => tmp1 = float, print(tmp1) shows tmp1 = 112 (no decimal).
>>>
>>> If I then calculate tmp2 as:
>>> tmp2 = tointeger( ( tmpV(1) - tmpV(0) ) / dvar )
>>> => tmp2 = integer, print(tmp2) shows tmp2 = 111.
>>>
>>> Why do I loose a unit here? tmp2 should be equal to 112 since
>>> tmp1 doesn't have any decimal, no?
>>> I need tmp2 to be an integer because it is used with ispan to
>>> define that new array I mentioned above.
>>>
>>> The on-line documentation specifies that the tointeger()
>>> function will truncate the decimals of a float- or
>>> double-type variable it is given. However, since I don't
>>> appear to have any decimal, why does it look like some
>>> truncation is taking place?
>>>
>>> Emilie
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Jan 6 16:18:44 2011

This archive was generated by hypermail 2.1.8 : Thu Jan 13 2011 - 09:24:21 MST