Re: Script crashes while computing WRF diagnostics (SLP, MDBZ, TD2)

Hi Joseph,

I'm guessing you are simply running out of memory. Once you are done with a do loop,
you might try deleting any variables you don't need before you proceed to the next do loop.

Also, have you tried getting rid of the "do" loops and using "-1" as the last argument to
wrf_user_getvar? This allows you to read in all time steps for that variable without
having to loop.

What I'm thinking about may look like your third attempt, except without the do loop:

> ncf->slp = wrf_user_getvar(a,"slp",-1) ;SLP
> ncf->mdbz = wrf_user_getvar(a,"mdbz",-1) ;MDBZ
> ncf->td2 = wrf_user_getvar(a,"td2",-1) ;TD2

If you predefine the NetCDF file that you're writing to, this will help your code be far more efficient,
especially if you are writing a lot of data to the file.

Please see our examples page on this:

http://www.ncl.ucar.edu/Applications/method_2.shtml

Let me know if this doesn't help and we'll try something else.

--Mary

On Jan 20, 2011, at 2:35 PM, Joseph Zambon wrote:

> NCL Users,
>
> I have a script that runs through a wrfout file and pulls out diagnostics as well as some variables I need to use. The variables are removed without a problem, however when I run this script for larger domains, the diagnostics appear to crash the system. I've been running this on a serial compute node as I've crashed the login node a few times (to the aggravation of other users, I'm sure) while trying to debug this.
>
> Attached is my script as it is right now (ncl_out.ncl). I've tried a coupled of different approaches to running the diagnostics for a domain that is 379(w_e) x 514 (s_n) and 673 timesteps. Using the diagnostic SLP and MDBZ as examples...
>
> Version 1:
> Original use was to create an array (ar_3d), save SLP to array at every time step, save array to netCDF file, then overwrite the array for the next variable (MDBZ). This way I was only using one large array (379x514x673) to pull out the variables. This script would crash after ~120 timesteps (not sure exactly where as it doesn't save the array until the end and would not print out the times right before dying). This way was probably the least efficient use of memory.
> *********************************************************************************
> ;create diagnostics array
> ar_3d = new( (/ ntimes, south_north, west_east /), float)
>
> ;Diagnostics
> do time = 0,ntimes-1,1 ;SLP
> slp = wrf_user_getvar(a,"slp",time) ; Sea level pressure in hPa
> ar_3d(time,:,:)=slp
> print(time)
> end do
> ncf->slp = ar_3d
>
> do time = 0,ntimes-1,1 ;MDBZ
> mdbz = wrf_user_getvar(a,"mdbz",time) ; Simulated radar refl.
> ar_3d(time,:,:)=mdbz
> print(time)
> end do
> ncf->mdbz = ar_3d
> *********************************************************************************
>
> Version 2:
> I assumed that this array was getting too large and was choking my memory. My next attempt was to save the output directly to the netCDF file and not use an array in the middle. This script would crash after 234 timesteps (the netCDF file wrote 234 frames and then died while processing the 235th).
> *********************************************************************************
> ;compute diagnostics and export to netCDF file
> do time = 0,ntimes-1,1 ;SLP
> ncf->slp(time,:,:) = (/wrf_user_getvar(a,"slp",time)/)
> print(time)
> end do
>
> ;compute diagnostics and export to netCDF file
> do time = 0,ntimes-1,1 ;MDBZ
> ncf->mdbz(time,:,:) = (/wrf_user_getvar(a,"mdbz",time)/)
> print(time)
> end do
> *********************************************************************************
>
> Version 3:
> My last attempt was to bunch all of the diagnostics within the same time loop. I'm not sure how efficient this is for I/O, it didn't seem to take much longer and it saved all 3 diagnostic variables to the netCDF file. This one stopped at 241 timesteps.
> *********************************************************************************
> ;compute diagnostics and export to netCDF file
> do time = 0,ntimes-1,1
> ncf->slp(time,:,:) = (/wrf_user_getvar(a,"slp",time)/) ;SLP
> ncf->mdbz(time,:,:) = (/wrf_user_getvar(a,"mdbz",time)/) ;MDBZ
> ncf->td2(time,:,:) = (/wrf_user_getvar(a,"td2",time)/) ;TD2
> print(time)
> end do
> *********************************************************************************
>
> If anyone has any advice on why my script might be crashing, it would be greatly appreciated. I'm chasing the path down memory allocation (should be a max of 4Gb per node), although I may be way off...
>
> CPU time : 1882.70 sec.
> Max Memory : 3256 MB
> Max Swap : 11190 MB
>
> Max Processes : 10
> Max Threads : 12
>
> Also attached is the text outfile (ncl.out) from my last analysis, they're all more or less the same (except the length of analysis). As shown in the outfile, I'm using NCL v. 5.2.0.
>
> Of course, any general critiques on how to make this NCL script more efficient would also be greatly appreciated.
>
> Please let me know if I neglected to mention anything important. Thanks!
>
> -Joe
>
>
>
> Joseph B. Zambon
> jbzambon@ncsu.edu
> NC State University
> Department of Marine, Earth and Atmospheric Sciences
>
>
> <ncl_out.ncl.txt>
>
> <ncl.out>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Jan 21 10:57:10 2011