Re: A truncation issue with tointeger()?

No problem :-). The truncation doesn't bother me as such, it is why it occurs in this particular case that does.

The page you refer to says that "If the input variable is float or double, then any fractional part of the input values is truncated. The dimensionality of the output matches that of the input.".

So, if my number is 112, there is no fractional part and there should not be any truncation and tointeger should return 112 (which it doesn't). If my number is 111.9999999, then the 0.999999 will be truncated and tointeger should return 111.

Emilie

On Jan 6, 2011, at 15:55, James Correia wrote:

> I can replicate what you did. And then I did a:
> print(numeric2int(g,1)) and the answer was 112 (no decimal)
> print(numeric2int(g,0)) and the answer was 111 (no decimal)
>
> In the function description of numeric2int option 1 is rounded, option 0 is truncated. so perhaps tointeger is truncated.
>
> http://www.ncl.ucar.edu/Document/Functions/Built-in/tointeger.shtml
>
> Indeed it says in the documentation that it is truncated.
>
> Pardon my guessing in the previous email
> jimmyc
>
> On Thu, Jan 6, 2011 at 4:46 PM, Emilie Vanvyve <evanvyve@ucar.edu> wrote:
> Hi James,
> I am not sure I understand what you mean. My 112 value is from a float variable. I would have expected it to display as 11.99999999999999 or something if the calculation wasn't leading to an exact 112 (which it should, but who knows) and I would then understand why tointeger() gives me 111 (the .99999999 part being then chopped).
>
> Emilie
>
> On Jan 6, 2011, at 15:43, James Correia wrote:
>
>> I don't know if this is related or not, but numeric types dont have the decimal, while float and integer do. It looks like tointeger must do rounding, but I am just guessing.
>> jimmyc
>>
>> On Thu, Jan 6, 2011 at 4:27 PM, Emilie Vanvyve <evanvyve@ucar.edu> wrote:
>> Hello,
>>
>> I just bumped into the below problem in my script and I cannot figure out why it happens. I was hoping someone would have an idea...
>>
>> I have the following 2 float variables:
>> tmpV = (/ -1.05, 15.75 /) ; lower and upper bounds of a new array to be later defined with a dvar increment
>> dvar = 0.15
>>
>> I calculate tmp1 as:
>> tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
>> => tmp1 = float, print(tmp1) shows tmp1 = 112 (no decimal).
>>
>> If I then calculate tmp2 as:
>> tmp2 = tointeger( ( tmpV(1) - tmpV(0) ) / dvar )
>> => tmp2 = integer, print(tmp2) shows tmp2 = 111.
>>
>> Why do I loose a unit here? tmp2 should be equal to 112 since tmp1 doesn't have any decimal, no?
>> I need tmp2 to be an integer because it is used with ispan to define that new array I mentioned above.
>>
>> The on-line documentation specifies that the tointeger() function will truncate the decimals of a float- or double-type variable it is given. However, since I don't appear to have any decimal, why does it look like some truncation is taking place?
>>
>> Emilie
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>> --
>> ------------------------------------------------------------------------------------------
>> The views expressed in this email do not necessarily reflect those of NOAA, the National Weather Service, or the University of Oklahoma.
>> ------------------------------------------------------------------------------------------
>> James Correia Jr.
>> OU CIMMS Research Associate
>> SPC HWT Liaison
>>
>
>
>
>
>
> --
> ------------------------------------------------------------------------------------------
> The views expressed in this email do not necessarily reflect those of NOAA, the National Weather Service, or the University of Oklahoma.
> ------------------------------------------------------------------------------------------
> James Correia Jr.
> OU CIMMS Research Associate
> SPC HWT Liaison
>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Jan 6 16:06:39 2011