Re: Segmentation Fault when opening lot of files....

From: Mary Haley <haley_at_nyahnyahspammersnyahnyah>
Date: Fri Jan 29 2010 - 18:22:21 MST

Hi Sebi,

Thanks for the additional information, it helps narrow down the problem.

The first thing to note is that NCL cannot handle variables that are
larger than (2**31)-1 in size (2147483647 bytes).

Considering that you have 4384 files, and if "hrf" is the same size on
every file, then the largest size that "hrf" can be on each file can
be is 2147483647 / 4384, which is roughly 490,000 bytes. This would
be the equivalent of about 122,400 floats, or 61,230 doubles. This is
the equivalent of about a 350 x 350 float array or a 247 x 247 double
array.

You didn't say how big "hrf" was on each file, but you will need to
make sure that the product of all them is not bigger than 2147483647
bytes.

What do you need to do with T once you read it all in? We might be
able to help you work around this issue.

--Mary

On Jan 29, 2010, at 4:16 AM, Sabeerali(sebi) wrote:

>
>
>
>
>
> Hi Mary,
>
> ---I have 4384 daily files all have same dimensions (lat, lon)
> ;---each having size of 2.25 MB
> I have used addfiles. The scripts works on fewer files...
>
>
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
> begin
> setfileoption("nc","SuppressClose",False)
> all_files = systemfunc ("ls *.nc")
> fall = addfiles (all_files, "r")
> ListSetType (fall, "join")
> T = fall[:]->hrf
> print(dimsizes(T))
> end
>
> Thanking you
>
> 2010/1/28 Mary Haley <haley@ucar.edu>
>
> Sebi,
>
> Can you be more specific about your problem? How big are the files
> you are trying to open, what variables are you trying to read, etc?
> Does the problem happen when you are using addfiles, or addfile on
> individual files?
>
> It would help if you could show us a script, explain where the error
> is occuring, how big the files are, whether the script works on
> fewer files, etc.
>
> --Mary
>
> On Jan 25, 2010, at 2:46 AM, Sabeerali(sebi) wrote:
>
>
> When I open lot of files it shows segmentaion fault.. Before opening
> files I have given setfieoption("nc","SuppressClose",False). I have
> read in FAQ that the general limit is around 1024 files. I am having
> 4000 files.. Is there any way in ncl to open that?
>
> Thanking you
>
>
>
> --
> **********************************
> Sebi
> Climate and Global Modeling Division
> Indian Institute of Tropical Meteorology
> Pashan, Pune, 411 008
> ****************************************
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
>
> --
> **********************************
> Sebi
> Climate and Global Modeling Division
> Indian Institute of Tropical Meteorology
> Pashan, Pune, 411 008
> ****************************************
>
>
>
> --
> **********************************
> Sebi
> Climate and Global Modeling Division
> Indian Institute of Tropical Meteorology
> Pashan, Pune, 411 008
> ****************************************
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Fri Jan 29 18:22:27 2010

This archive was generated by hypermail 2.1.8 : Mon Feb 01 2010 - 08:05:34 MST