Re: read from 239 netCDF files and write to only one netCDF file again

From: Dennis Shea (shea AT XXXXXX)
Date: Mon Jan 07 2002 - 11:05:22 MST

  • Next message: Mary Haley: "Re: external functions on IBM AIX"

    Certainly looks like a hardware limitation [memory]
    on your machine. I'm not sure that this can be solved
    in any direct way.

    For this type of problem I would recommend using the
    netCDF operators (NCO) if they are available on your system.
    These are public domain "operators" that work on bot
    netCDF and HDF4 files. If they are available try:

    ncrcat -v CLOUD 19??-??.nc CLOUD.nc

    If this fails, you may have to create several files
    containing "CLOUD"

    ---
    Even if this were to work .... how do u plan to read
    all the data into your machine's memory?
    ---
    

    Incidently, in contributed.ncl there is a function "addfiles_GetVar" that can be used to make the NCL code a bit cleaner. This function adds the attributes and coordinate variables.

    begin diri = "/disk/cj2/ccm/ccm-mqz/run.amip2/" ; directory where fil es reside fils = systemfunc ("ls "+diri+"19??-??.nc") f = addfiles (fils, "r") ; note the "s" of addfile

    ListSetType (f, "cat") ; concatenate or "merge" T = addfiles_GetVar (f, fils, "CLOUD" )

    filo = "./nc/CLOUD.nc" system ("/usr/bin/rm "+filo) ; remove any pre-existing file fo = addfile(filo , "c") ; open output file fo@title = "CLOUD from model files" fo->CLOUD = T ; write CLOUD to a file end

    >Date: Mon, 7 Jan 2002 11:20:44 -0600 (CST) >From: Qiaozhen Mu <qiaozhen@utig.ig.utexas.edu> >Subject: read from 239 netCDF files and write to only one netCDF file again >To: ncl-talk@UCAR.EDU >MIME-Version: 1.0 >Content-MD5: ZWYy/5qhSGifvmjPlUviWQ== > >Dear all hands, > >Thanks for Dennis Shea's kind help. I have used the ncl to get fields out from >many history files. But now I meet another question. I try to get the field >CLOUD out from 239 history files. There are 18 levels, 128 longitudes and 64 >latitudes for each CLOUD field in each history file. So when I ran the ncl, it >reminded me that > >""""" >fatal:NclMalloc Failed:[errno=11]:Resource temporarily unavailable >fatal:_NclBuildConcatArray : Memory allocation failure > >fatal:Execute: Error occurred at or near line 6 >""""" > >There isn't enough memory to deal with it. What can I do? The following is the >CLOUD.ncl I used to get the CLOUD out. > > > begin > diri = "/disk/cj2/ccm/ccm-mqz/run.amip2/" ; directory where fil >es reside > fils = systemfunc ("ls "+diri+"19??-??.nc") > f = addfiles (fils, "r") ; note the "s" of addfile > > ListSetType (f, "cat") ; concatenate or "merge" > T = f[:]->CLOUD > print (dimsizes(T)) ; T will have 60 time steps > T!0 = "time" > T!1 = "lev" > T!2 = "lat" > T!3 = "lon" > T&time = f[:]->time ; time coord variable > T&lev = f[0]->lev ; get lev from the 1st file > T&lat = f[0]->lat ; get lat from the 1st file > T&lon = f[0]->lon ; get lon from the 1st file > > filo = "./nc/CLOUD.nc" > system ("/usr/bin/rm "+filo) ; remove any pre-existing file > fo = addfile(filo , "c") ; open output file > fo@title = "CLOUD from model files" > > fo->CLOUD = T ; write CLOUD to a file > end > >Thanks in advance. > >Best regards, >Qiaozhen > >*************************************** >*The University of Texas at Austin * >*Institute for Geophysics * >*4412 Spicewood Springs Rd., Bldg. 600* >*Austin, Texas 78759-8500 * >*phone:(512) 471-0462 * >*************************************** >



    This archive was generated by hypermail 2b29 : Mon Jan 07 2002 - 11:41:33 MST