Strange behavior in algebric computations

From: Mateus Teixeira <mateus.teixeira_at_nyahnyahspammersnyahnyah>
Date: Thu, 1 Oct 2009 11:13:09 -0300

Dear NCL users,

I'm doing some algebric computations that are having strange behavior
depending on way the formula is written. When written this way

su = su + dh * ( u(c,t,k-1,j,i) + u(c,t,k,j,i) ) / 2

I'm getting a missing value (the formula is inside a do loop), as shown below:

(0) sv(before)=0; dh=1271; v(k-1)=0.699997; v(k)=-4.8
(0) sv(after)=-2605.55
(0) sv(before)=-2605.55; dh=2114; v(k-1)=-4.8; v(k)=-10.7
(0) sv(after)=-32767 <= MISSING VALUE

But when the formula is written as

su = su + 0.5 * dh * ( u(c,t,k-1,j,i) + u(c,t,k,j,i) )

I haven't a missingl value, as shown below for the same stop point showed above

(0) sv(before)=0; dh=1271; v(k-1)=0.699997; v(k)=-4.8
(0) sv(after)=-2605.55
(0) sv(before)=-2605.55; dh=2114; v(k-1)=-4.8; v(k)=-10.7
(0) sv(after)=-18989.1 <=CORRECT VALUE

I made these same computations by hand in NCL with both formulae and
get the correct value given by the second way of the formula.

Can anyone imagine what is causing this? I thought that the float
representation of the 1/2 could be the source of the error, but the
difference between missing value and the correct value is not little.

Best regards,

-- 
Mateus da Silva Teixeira
Meteorologista
Instituto de Pesquisas Meteorológicas - IPMet
Universidade Estadual Paulista - UNESP
Av. Luis Edmundo Carrijo Coube, 14-01 - Cx. Postal 281 - CEP 17033-360
Bauru - SP - Brasil
Fone: +55 14 3103-6030 / fax: 3203-3649
Registered Linux User #466740 (http://counter.li.org/)
_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Oct 01 2009 - 08:13:09 MDT

This archive was generated by hypermail 2.2.0 : Mon Oct 05 2009 - 13:28:34 MDT