Re: large netcdf file/ failure copying attributes and variables

See the "Largefile" option:
http://www.ncl.ucar.edu/Document/Functions/Built-in/setfileoption.shtml
I wonder why you are using ncl to copy a file though? I would do this in
a SHELL script (e.g., bash), using cp to copy the file and NCO tools to
sub-set. It looks like you could even run NCO pointing to the file(s) on
the remote server and write to you local machine on the fly, and no need
to even copy?
Cheers,
Jatin.


On 10/06/14 9:26 AM, Sam McClatchie (NOAA Federal) wrote:
> Colleagues
>
> I was trying to copy from a large (6.5GB) netcdf file on a remote
> server to my local disk (prior to subsetting) in order to explore a
> ROMS model output. I wanted to keep the local data as a netcdf file,
> so I borrowed some example NCL code to copy the attributes and
> variables from the remote server to my local file. All went well until
> the local file size reached 2.3 GB. The error message returned was
> "fatal:NetCDF: Operation not allowed in define mode: error attempting
> to write variable (temp_obc) to file
> (../data/roms_UCSC_31yr/wc12_ccsra31_his_19800109_19800117.nc)".
>
> Here is the code:
>
> begin
> remotedir =
> "http://oceanmodeling.pmc.ucsc.edu:8080/thredds/dodsC/wc12.0_ccsra31_01/posterior/"
> ; 31 year reanalysis after data assimilation
> basename = "wc12_ccsra31_his" ; base of filenames
> start = 19800109 ;first date in the series
> last = 19800117 ; test with only 1 file
> ; last = 20101221
> overlap = 4 ; files overlap by 4 days
> int = 8; files span 8 day interval
>
> ;;; loop through all of the files one at a time
> i = 1
> t1 = start
> t2 = start + int
> files = 1 ; test
>
> do while(i.le.files)
> remotefile = remotedir + basename + "_" + t1 + "_" + t2 + ".nc"
> print(remotefile)
>
> r_ncdf = addfile(remotefile,"r") ; read only from remote server
> vars = getfilevarnames(r_ncdf)
> print(vars)
>
> ;; write variables to local netcdf file
> localfile = "../data/roms_UCSC_31yr/" + basename + "_" + t1 + "_"
> + t2 + ".nc"
> system("/bin/rm -f " + localfile) ; remove any pre-existing file
> l_ncdf = addfile(localfile, "c"); create a new reference for a
> local netcdf file
>
> att_names = getvaratts(r_ncdf) ; get remote file's global
> attributes
> print(att_names)
> if(.not.all(ismissing(att_names))) then
> do i = 0,dimsizes(att_names)-1
> print("copy_fileatts: global attributes->" + att_names(i))
> l_ncdf@$att_names(i)$ = r_ncdf@$att_names(i)$ ; copy
> remote file's global attributes
> end do
> end if
>
> names = getfilevarnames(r_ncdf) ; get remote file's variable
> names and print them
> print (names)
>
> ; loop over variables and copy to netCDF file
>
> do i=0, dimsizes(names)-1
> print("writing local ncdf: i,name="+ i+ " "+ names(i))
> l_ncdf->$names(i)$ = r_ncdf->$names(i)$
> end do
>
> t1 = t1 + overlap
> t2 = t2 + overlap
> i = i + 1
> end do
>
> print("Files processed = " + files)
> end
> -------------------------------------------------------------
> Any ideas what went wrong here?
>
> Thanks
>
> Sam
> --
> email signature Sam McClatchie,
> Supervisory oceanographer,
> Fisheries oceanography, Ichthyoplankton, Ship Operations
> Southwest Fisheries Science Center, NOAA,
> 8901 La Jolla Shores Dr.
> La Jolla, CA 92037-1509, U.S.A.
> email <Sam.McClatchie@noaa.gov>
> Office: 858 546 7083, Cellular: 858 752 8495
> Research home page <www.fishocean.info>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk


Received on Tue Jun 10 12:46:20 2014