Re: A truncation issue with tointeger()?

Oh... Is there some kind of formatting/rounding going on with print() then? I thought that using that function would give me the exact value of the variable (111.99999 in this case then), especially since it sometimes likes displaying numbers with a fractional part I don't expect (see my other e-mail about ispan).

If I can't trust that ( tmpV(1) - tmpV(0) ) / dvar = 112 exactly (some other time, it does though), I never can expect a float to convert to an integer without any similar problem. Pushing it to the limit, I could have tointeger((3.0-1.0)/1.) not equal to 2 but 1... Should I then always use the round() function except if I want a truncation to occur?

Emilie

On Jan 6, 2011, at 16:18, Dennis Shea wrote:

> Rounding issue.
>
> tmpV = (/ -1.05, 15.75 /)
> dvar = 0.15
> tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
> print(tmp1) ; 112
> print(sprintf("%5.2f", tmp1)) ; 112.00
> print(sprintf("%12.5f", tmp1)) ; 111.99999
>
> i1 = toint(tmp1) ; 111
> i2 = floattoint(tmp1) ; 111
> i3 = round(tmp,3) ; 112
>
> On 01/06/2011 04:06 PM, Emilie Vanvyve wrote:
>> No problem :-). The truncation doesn't bother me as such, it is why it
>> occurs in this particular case that does.
>>
>> The page you refer to says that /"If the input variable is float or
>> double, then any fractional part of the input values is truncated. The
>> dimensionality of the output matches that of the input."/.
>>
>> So, if my number is 112, there is no fractional part and there should
>> not be any truncation and tointeger should return 112 (which it
>> doesn't). If my number is 111.9999999, then the 0.999999 will be
>> truncated and tointeger should return 111.
>>
>> Emilie
>>
>> On Jan 6, 2011, at 15:55, James Correia wrote:
>>
>>> I can replicate what you did. And then I did a:
>>> print(numeric2int(g,1)) and the answer was 112 (no decimal)
>>> print(numeric2int(g,0)) and the answer was 111 (no decimal)
>>>
>>> In the function description of numeric2int option 1 is rounded, option
>>> 0 is truncated. so perhaps tointeger is truncated.
>>>
>>> http://www.ncl.ucar.edu/Document/Functions/Built-in/tointeger.shtml
>>>
>>> Indeed it says in the documentation that it is truncated.
>>>
>>> Pardon my guessing in the previous email
>>> jimmyc
>>>
>>> On Thu, Jan 6, 2011 at 4:46 PM, Emilie Vanvyve <evanvyve@ucar.edu
>>> <mailto:evanvyve@ucar.edu>> wrote:
>>>
>>> Hi James,
>>> I am not sure I understand what you mean. My 112 value is from a
>>> float variable. I would have expected it to display as
>>> 11.99999999999999 or something if the calculation wasn't leading
>>> to an exact 112 (which it should, but who knows) and I would then
>>> understand why tointeger() gives me 111 (the .99999999 part being
>>> then chopped).
>>>
>>> Emilie
>>>
>>> On Jan 6, 2011, at 15:43, James Correia wrote:
>>>
>>>> I don't know if this is related or not, but numeric types dont
>>>> have the decimal, while float and integer do. It looks like
>>>> tointeger must do rounding, but I am just guessing.
>>>> jimmyc
>>>>
>>>> On Thu, Jan 6, 2011 at 4:27 PM, Emilie Vanvyve <evanvyve@ucar.edu
>>>> <mailto:evanvyve@ucar.edu>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I just bumped into the below problem in my script and I
>>>> cannot figure out why it happens. I was hoping someone would
>>>> have an idea...
>>>>
>>>> I have the following 2 float variables:
>>>> tmpV = (/ -1.05, 15.75 /) ; lower and upper bounds of a new
>>>> array to be later defined with a dvar increment
>>>> dvar = 0.15
>>>>
>>>> I calculate tmp1 as:
>>>> tmp1 = ( tmpV(1) - tmpV(0) ) / dvar
>>>> => tmp1 = float, print(tmp1) shows tmp1 = 112 (no decimal).
>>>>
>>>> If I then calculate tmp2 as:
>>>> tmp2 = tointeger( ( tmpV(1) - tmpV(0) ) / dvar )
>>>> => tmp2 = integer, print(tmp2) shows tmp2 = 111.
>>>>
>>>> Why do I loose a unit here? tmp2 should be equal to 112 since
>>>> tmp1 doesn't have any decimal, no?
>>>> I need tmp2 to be an integer because it is used with ispan to
>>>> define that new array I mentioned above.
>>>>
>>>> The on-line documentation specifies that the tointeger()
>>>> function will truncate the decimals of a float- or
>>>> double-type variable it is given. However, since I don't
>>>> appear to have any decimal, why does it look like some
>>>> truncation is taking place?
>>>>
>>>> Emilie

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Jan 6 16:28:50 2011