large netcdf file/ failure copying attributes and variables

From: Sam McClatchie (NOAA Federal) <sam.mcclatchie_at_nyahnyahspammersnyahnyah>
Date: Mon Jun 09 2014 - 17:26:21 MDT

Colleagues

I was trying to copy from a large (6.5GB) netcdf file on a remote server
to my local disk (prior to subsetting) in order to explore a ROMS model
output. I wanted to keep the local data as a netcdf file, so I borrowed
some example NCL code to copy the attributes and variables from the
remote server to my local file. All went well until the local file size
reached 2.3 GB. The error message returned was "fatal:NetCDF: Operation
not allowed in define mode: error attempting to write variable
(temp_obc) to file
(../data/roms_UCSC_31yr/wc12_ccsra31_his_19800109_19800117.nc)".

Here is the code:

begin
   remotedir =
"http://oceanmodeling.pmc.ucsc.edu:8080/thredds/dodsC/wc12.0_ccsra31_01/posterior/"
; 31 year reanalysis after data assimilation
   basename = "wc12_ccsra31_his" ; base of filenames
   start = 19800109 ;first date in the series
   last = 19800117 ; test with only 1 file
   ; last = 20101221
   overlap = 4 ; files overlap by 4 days
   int = 8; files span 8 day interval

   ;;; loop through all of the files one at a time
   i = 1
   t1 = start
   t2 = start + int
   files = 1 ; test

   do while(i.le.files)
     remotefile = remotedir + basename + "_" + t1 + "_" + t2 + ".nc"
     print(remotefile)

     r_ncdf = addfile(remotefile,"r") ; read only from remote server
     vars = getfilevarnames(r_ncdf)
     print(vars)

    ;; write variables to local netcdf file
     localfile = "../data/roms_UCSC_31yr/" + basename + "_" + t1 + "_" +
t2 + ".nc"
     system("/bin/rm -f " + localfile) ; remove any pre-existing file
     l_ncdf = addfile(localfile, "c"); create a new reference for a
local netcdf file

    att_names = getvaratts(r_ncdf) ; get remote file's global
attributes
    print(att_names)
       if(.not.all(ismissing(att_names))) then
         do i = 0,dimsizes(att_names)-1
           print("copy_fileatts: global attributes->" + att_names(i))
           l_ncdf@$att_names(i)$ = r_ncdf@$att_names(i)$ ; copy
remote file's global attributes
         end do
       end if

   names = getfilevarnames(r_ncdf) ; get remote file's variable
names and print them
   print (names)

; loop over variables and copy to netCDF file

       do i=0, dimsizes(names)-1
         print("writing local ncdf: i,name="+ i+ " "+ names(i))
         l_ncdf->$names(i)$ = r_ncdf->$names(i)$
       end do

     t1 = t1 + overlap
     t2 = t2 + overlap
     i = i + 1
   end do

     print("Files processed = " + files)
   end
-------------------------------------------------------------
Any ideas what went wrong here?

Thanks

Sam
--
email signature Sam McClatchie,
Supervisory oceanographer,
Fisheries oceanography, Ichthyoplankton, Ship Operations
Southwest Fisheries Science Center, NOAA,
8901 La Jolla Shores Dr.
La Jolla, CA 92037-1509, U.S.A.
email <Sam.McClatchie@noaa.gov>
Office: 858 546 7083, Cellular: 858 752 8495
Research home page <www.fishocean.info>



_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk


Received on Mon Jun 09 11:26:38 2014

This archive was generated by hypermail 2.1.8 : Wed Jul 23 2014 - 15:33:46 MDT