Re: Running NCL batch job in Bluevista?

> I need to process a large amount of CCSM files for simple statistical
> analysis and graph later on. At this stage, I thought
> of NCL, owing to its simplicity in handling netcdf files. After some days
> of file movement I realized that it may be better for me to process CCSM
> files in NCAR Bluevista/Bluesky instead of transfering them back to Hawaii
> where I work.

NCAR "tempest" is meant for data processing.
It has 128 processors and 32 or 64 GB of memory [I forget]

>However, since the total size of files is quite big (>50 Gb),
>I have a number of questions in mind:
>
> 1) running my NCL script would take quite some time, therefore is it
> possible to run NCL in a parallel mode

in "parallel" mode: no

> 2) or submit it as a batch job?
> if so, could you please share your batch job script or if possible
> point to the sample job script format used by SCD?

yes. It is possible to use a batch script.
However, I have not done this so I can not give an example

People frequently use:

     ncl foo.ncl >&! foo.out &

However, the batch approach will allow you to select a lower
cost queue perhaps.

> 3) Do you ever use NCL to directly access files in MSS, and if so could
> you please share your scripts?

Yes. The following access the MSS and uses the netCDF operator
      ncrcat to extract a variable from a set of netCDF files
      The system statemenent are commented. Yoe should make sure
      the msscmd, ncocmd and rmcmd are correct before activating.

begin
   nyrStrt = 300
   nyrLast = 999

   mssi= "/CCSM/csm/b20.007/ocn/hist/" ; directory
   fili= "b20.007.pop.h.0" ; first part of file name
   diri= "/ptmp/shea/" ; where to put msrcp data (input)
   diro= "/ptmp/shea/output/" ; where to put ncrcat data (outout)
   filo= "b20.TEMP." ; rename output files (root name)

   do nyear=nyrStrt,nyrLast ; each year has 12 files

       msscmd= "msrcp -n 'mss:" + mssi + fili + nyear + "-[0-1][0-9].nc' "+ diri+"."
       print (" msscmd= "+ msscmd)
      ;system (msscmd)

       ncocmd= "ncrcat -v TEMP " + diri + fili + "*.nc "+diro+filo+nyear+".nc"
       print ("ncocmd = "+ncocmd)
      ;system (ncocmd)

       rmcmd = "'rm' "+diri+fili+nyear+"*.nc"
       print ("rmcmd ="+ rmcmd)
      ;system (rmcmd)
   end do

end

> 4) If there is other better alternative, please kindly Bsuggest.
> _______________________________________________
_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Aug 04 2006 - 07:24:03 MDT