Re: how to extract data from a huge nc data file?

Hi,

Perhaps CDO can help? (http://www.mpimet.mpg.de/~cdo/)

--- En date de : Mar 24.6.08, Dennis Shea <shea_at_ucar.edu> a écrit :
De: Dennis Shea <shea_at_ucar.edu>
Objet: Re: [ncl-talk] how to extract data from a huge nc data file?
À: "Zhang Tao" <Tao.Zhang_at_noaa.gov>
Cc: ncl-talk_at_ucar.edu
Date: Mardi 24 Juin 2008, 0h18

[1] NCL can handle files greater than 4GB if large file support is
available.

[2] Currently, NCL can not handle individual variables > 2GB.
      The reason is that 32-bit addressing does not support this size.

[3] I am not sure if the NCO [eg: ncks] support variables > 2GB
      Even with the ncks below you would need >2GB of memory
      600*17*320*395*4 bytes.

Good luck

Zhang Tao wrote:
> I am using Mac OS version 10.5.2 and processor is Dual 2 GHz PowerPC
> G5. Memory is 2.5GB DDR SDRAM.
> Is this a 32-bit OS? What's a 64-bit machine?
> Thanks,
> Tao
>
> Xianglei Huang wrote:
>> ncks is the right command for this type of job. It seems to me
>> that most likely you are using a 32-bit OS so it cannot handle netcdf
>> files beyond 4G (or 2G?). Try it on a 64-bit machine and it should
work.
>>
>> Zhang Tao wrote:
>>> Hi, all,
>>>
>>> I have a huge netcdf data file
>>> "thetao_O1.20C3M_1.CCSM.ocnm.1950-01_cat_1999-12.nc" and
>>> the size is about 12G.
>>>
>>> I want to extract parts of data and generate a small size netcdf

>>> file, the specific command is listed in point 3 below
>>> and I get the error message "segmentation fault" see
point 2 below.
>>> I can use "ncdump" to look at the attribution of the
data, see
>>> point 1 below.
>>>
>>> Does anyone know how to extract data from this huge nc file?
>>>
>>> Thanks,
>>> Tao
>>>
>>> 1)
>>> netcdf thetao_O1.20C3M_1.CCSM.ocnm.1950-01_cat_1999-12 {
>>> dimensions:
>>> lon = 320 ;
>>> lat = 395 ;
>>> depth = 40 ;
>>> bnds = 2 ;
>>> time = UNLIMITED ; // (600 currently)
>>> variables:
>>> double lon_bnds(lon, bnds) ;
>>> double lat_bnds(lat, bnds) ;
>>> double depth_bnds(depth, bnds) ;
>>> double time_bnds(time, bnds) ;
>>> double lon(lon) ;
>>> lon:axis = "X" ;
>>> lon:standard_name = "longitude" ;
>>> lon:bounds = "lon_bnds" ;
>>> lon:long_name = "Longitude" ;
>>> lon:units = "degrees_east" ;
>>> double lat(lat) ;
>>> lat:axis = "Y" ;
>>> lat:standard_name = "latitude" ;
>>> lat:bounds = "lat_bnds" ;
>>> lat:long_name = "Latitude" ;
>>> lat:units = "degrees_north" ;
>>> double depth(depth) ;
>>> depth:axis = "Z" ;
>>> depth:standard_name = "depth" ;
>>> depth:positive = "down" ;
>>> depth:units = "m" ;
>>> depth:bounds = "depth_bnds" ;
>>> double time(time) ;
>>> time:calendar = "noleap" ;
>>> time:standard_name = "time" ;
>>> time:axis = "T" ;
>>> time:units = "days since 0000-1-1" ;
>>> time:bounds = "time_bnds" ;
>>> float thetao(time, depth, lat, lon) ;
>>> thetao:comment = "Created using NCL code
>>> CCSM_ocnm_2cf.ncl on\n",
>>> " machine mineral" ;
>>> thetao:missing_value = 1.e+20f ;
>>> thetao:long_name =
"sea_water_potential_temperature" ;
>>> thetao:cell_methods = "time: mean (interval: 1
month)" ;
>>> thetao:history = "Interpolated to regular grid
from
>>> dipole grid,\n",
>>> "TEMP+273.15" ;
>>> thetao:units = "K" ;
>>> thetao:original_units = "C" ;
>>> thetao:original_name = "TEMP" ;
>>> thetao:standard_name =
>>> "sea_water_potential_temperature" ;
>>> thetao:_FillValue = 1.e+20f ;
>>>
>>> 2)
>>> ---------------------------
>>> /nas/tzhang/ZTIPCC4/ncar_ccsm3_0/20c3m/nc>
>>> extract_thetao_20C3M_1.CCSM.csh &
>>> [1] 41886
>>> /nas/tzhang/ZTIPCC4/ncar_ccsm3_0/20c3m/nc> Segmentation fault
>>>
>>> [1] Exit 139
extract_thetao_20C3M_1.CCSM.csh
>>> ----------------------------------------------------
>>>
>>> 3)
>>> ncks -c -F -d depth,1,17 -v thetao
>>> thetao_O1.20C3M_1.CCSM.ocnm.1950-01_cat_1999-12.nc
>>> thetao_upper_O1.20C3M_1.CCSM.ocnm.1950-01_cat_1999-12.nc
>>>

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk _____________________________________________________________________________ Envoyez avec Yahoo! Mail. Une boite mail plus intelligente http://mail.yahoo.fr

_______________________________________________
ncl-talk mailing list
ncl-talk_at_ucar.edu
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Tue Jun 24 2008 - 00:50:42 MDT