Re: Slow plot processing

Hi Paul,
It is pretty difficult to glean anything useful from this backtrace. Probably more of it would not help much. In order to figure this out I would need a script and enough of your data to reproduce the seg fault here. If you can make this available I would certainly appreciate it. See the "Report Bugs" page under Support on the NCL web site for uploading instructions if you are not familiar with the procedure.
 -dave

On Dec 27, 2012, at 3:18 AM, Paul Scorer wrote:

> I am now using MeshFill in production. However, I am getting
> _occasional_ SegFaults and other crashes, the latest of which provided a
> backtrace, which I have copied below. Maybe this is useful?
>
> The memory map is very long: I can supply this is helpful.
>
> Regards
>
> PaulS
>
>
> *** glibc detected *** /opt/ncl-6.1.0/bin/ncl: free(): invalid next size
> (fast): 0x0000000026bcf880 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x373a478526]
> /opt/ncl-6.1.0/bin/ncl(NclFree+0xe)[0x67b261]
> /opt/ncl-6.1.0/bin/ncl(_NclPopFrame+0xd7)[0x72f307]
> /opt/ncl-6.1.0/bin/ncl(_NclFuncCallOp+0x97)[0x73378c]
> /opt/ncl-6.1.0/bin/ncl(CallFUNC_CALL_OP+0x4e)[0x7270c5]
> /opt/ncl-6.1.0/bin/ncl(_NclExecute+0x365)[0x72d675]
> /opt/ncl-6.1.0/bin/ncl(_NclFuncCallOp+0x34)[0x733729]
> /opt/ncl-6.1.0/bin/ncl(CallFUNC_CALL_OP+0x4e)[0x7270c5]
> /opt/ncl-6.1.0/bin/ncl(_NclExecute+0x365)[0x72d675]
> /opt/ncl-6.1.0/bin/ncl(_NclProcCallOp+0x34)[0x7337c7]
> /opt/ncl-6.1.0/bin/ncl(CallPROC_CALL_OP+0x4e)[0x72679f]
> /opt/ncl-6.1.0/bin/ncl(_NclExecute+0x405)[0x72d715]
> /opt/ncl-6.1.0/bin/ncl(_NclFuncCallOp+0x34)[0x733729]
> /opt/ncl-6.1.0/bin/ncl(CallFUNC_CALL_OP+0x4e)[0x7270c5]
> /opt/ncl-6.1.0/bin/ncl(_NclExecute+0x365)[0x72d675]
> /opt/ncl-6.1.0/bin/ncl(_NclProcCallOp+0x34)[0x7337c7]
> /opt/ncl-6.1.0/bin/ncl(CallPROC_CALL_OP+0x4e)[0x72679f]
> /opt/ncl-6.1.0/bin/ncl(_NclExecute+0x405)[0x72d715]
> /opt/ncl-6.1.0/bin/ncl(yyparse+0x82c)[0x614d5e]
> /opt/ncl-6.1.0/bin/ncl(main+0xbd0)[0x61082e]
> /lib64/libc.so.6(__libc_start_main+0xed)[0x373a42135d]
> /opt/ncl-6.1.0/bin/ncl(vfprintf+0x4c9)[0x60fb69]
> ======= Memory map: ========
> ... <snip> ...
>
>
>
> On Fri, 2012-12-21 at 12:02 -0700, David Brown wrote:
>> Hi Paul,
>> Thanks for the feedback. And happy holidays to you!
>> -dave
>>
>> On Dec 21, 2012, at 8:15 AM, Paul Scorer wrote:
>>
>>> Yahoo!!
>>>
>>> MeshFill reduced the time to plot 41 parameters from 42m33s to 5m6.6s!!!
>>>
>>> A rather significant improvement!
>>>
>>> Season's Greetings to All and Best Wishes for the new Year Number!
>>>
>>> Regards
>>>
>>> PaulS
>>>
>>> On Tue, 2012-12-18 at 17:43 -0700, David Brown wrote:
>>>> Hi Paul,
>>>> RasterFill plotting slows down with 2D coordinates. CellFill less so. The disadvantage of CellFill is mainly that it can create very large output files, although if you plot to a raster format such as png that is not an issue. The new algorithm that Dennis mentions helps to overcome the rasterfill slowdown with 2D coordinates. If you have the latest version of NCL you can try it out by setting cnFillMode to "MeshFill". Just remember this is a work in progress. It will fail miserably with non-rectangular grids, for instance.
>>>> -dave
>>>>
>>>>
>>>> On Dec 18, 2012, at 2:41 PM, Paul Scorer wrote:
>>>>
>>>>> Does the "new plotting algorithm" relate only to to pdf output, or is it
>>>>> more fundamental, and does it apply only to V6.1, a future release, ...?
>>>>>
>>>>> I have "slow plot processing" issues too. I changed AreaFill to
>>>>> RasterFill (under V6.0, with a very significant speed up). Are you
>>>>> saying that CellFill is faster than RasterFill, for _all_ o/p formats,
>>>>> for pdf, for ???
>>>>>
>>>>> If you want to test a troublesome case, let me know, and I can try and
>>>>> find a suitable data file, and make the code available.
>>>>>
>>>>> Many Thanks for your unrivalled service,
>>>>>
>>>>> Cheers
>>>>>
>>>>> PaulS
>>>>>
>>>>> On Fri, 2012-12-14 at 07:44 -0700, Dennis Shea wrote:
>>>>>> Replace
>>>>>> res@cnFillMode = "RasterFill"
>>>>>>
>>>>>> with
>>>>>> res@cnFillMode = "CellFill"
>>>>>>
>>>>>> Also, a new plotting algorithim is being tested,
>>>>>> please ftp the file to
>>>>>>
>>>>>> ftp ftp.cgd.ucar.edu
>>>>>> anonymous
>>>>>> email
>>>>>> cd incoming
>>>>>> put MYD09A1.A2007273.h03v07.005.2007285103507.hdf
>>>>>> quit
>>>>>>
>>>>>> Send an email after the successful transfer.
>>>>>>
>>>>>>
>>>>>> On 12/14/12 6:33 AM, Filippo Locci wrote:
>>>>>>> Hi,
>>>>>>> I'm working with this example from
>>>>>>> http://www.hdfeos.org/zoo/index_openLPDAAC_Examples.php#MOD
>>>>>>> <http://www.hdfeos.org/zoo/index_openLPDAAC_Examples.php#MOD>
>>>>>>>
>>>>>>> ;This example code illustrates how to access and visualize LP_DAAC_MYD
>>>>>>> Grid file in NCL.
>>>>>>> ;If you have any questions, suggestions, comments on this example,
>>>>>>> please use the HDF-EOS Forum (http://hdfeos.org/forums).
>>>>>>> ;If you would like to see an example of any other NASA HDF/HDF-EOS
>>>>>>> data product that is not listed in the HDF-EOS Comprehensive Examples
>>>>>>> page (http://hdfeos.org/zoo),
>>>>>>> ;feel free to contact us at eoshelp@hdfgroup.org or post it at the
>>>>>>> HDF-EOS Forum (http://hdfeos.org/forums).
>>>>>>>
>>>>>>> load "$NCARG_ROOT/lib/ncarg/nclex/gsun/gsn_code.ncl"
>>>>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"
>>>>>>> load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"
>>>>>>>
>>>>>>> ; This is an example of a 2-D grid file data field.
>>>>>>> ; It is assumed users know how to obtain information such as
>>>>>>> _FillValue from HDFView.
>>>>>>> ; For information about HDFView, visit
>>>>>>> http://www.hdfgroup.org/hdf-java-html/hdfview/.
>>>>>>>
>>>>>>> begin
>>>>>>>
>>>>>>> eos_file=addfile("MYD09A1.A2007273.h03v07.005.2007285103507.hdf",
>>>>>>> "r") ; Read file.
>>>>>>> he2_file = addfile("MYD09A1.A2007273.h03v07.005.2007285103507.hdf.he2",
>>>>>>> "r") ; Read file for the second time to efficiently obtain lat and
>>>>>>> lon.
>>>>>>>
>>>>>>> ; List available fields and attributes.
>>>>>>> ; print(eos_file)
>>>>>>>
>>>>>>> ; read data field
>>>>>>> data_raw = eos_file->sur_refl_b02
>>>>>>>
>>>>>>> ; print(data_raw)
>>>>>>>
>>>>>>> ; Apply scale factor according to the field attribute
>>>>>>> ; correct_data_value = (raw_data_value - offset) * scale_factor
>>>>>>> data = (data_raw - data_raw@add_offset) * data_raw@scale_factor
>>>>>>>
>>>>>>> data@_FillValue = (data_raw@_FillValue - data_raw@add_offset) *
>>>>>>> data_raw@scale_factor
>>>>>>>
>>>>>>> data@long_name = data_raw@long_name
>>>>>>> data@units = data_raw@units
>>>>>>>
>>>>>>> ; Associate longitude and latitude
>>>>>>> data@lat2d=he2_file->GridLat_MOD_Grid_500m_Surface_Reflectance
>>>>>>> data@lon2d=he2_file->GridLon_MOD_Grid_500m_Surface_Reflectance
>>>>>>>
>>>>>>> xwks=gsn_open_wks("pdf","MYD09A1.A2007273.h03v07.005.2007285103507_sur_refl_b02.ncl")
>>>>>>> ; open workstation
>>>>>>>
>>>>>>> res=True ; plot mods desired
>>>>>>> res@cnFillOn=True ; enable contour fill
>>>>>>> res@gsnMaximize=True; make plot large
>>>>>>> res@gsnPaperOrientation = "portrait" ; force portrait orientation
>>>>>>> res@cnLinesOn=False ; turn off contour lines
>>>>>>> res@cnLineLabelsOn = False; turn off contour line labels
>>>>>>> res@gsnSpreadColors = True ; use the entire color spectrum
>>>>>>> res@cnFillMode = "RasterFill" ; faster
>>>>>>> res@lbOrientation ="vertical" ;vertical labels
>>>>>>> res@cnMissingValFillPattern = 0 ; missing value pattern is set to "SolidFill"
>>>>>>> res@cnMissingValFillColor = 0; white color for missing values
>>>>>>>
>>>>>>> ; the following 5 sentences are used to create a zoomed image
>>>>>>> res@mpLimitMode = "LatLon"
>>>>>>> res@mpMinLatF = min(data@lat2d) ; Set limits of map, based on the
>>>>>>> min/max of the dataset latitude/longitude
>>>>>>> res@mpMaxLatF = max(data@lat2d) ;
>>>>>>> res@mpMinLonF = min(data@lon2d) ;
>>>>>>> res@mpMaxLonF = max(data@lon2d) ;
>>>>>>>
>>>>>>> gsn_define_colormap(xwks,"BlAqGrYeOrReVi200") ; choose colormap
>>>>>>> res@tiMainString = "MYD09A1.A2007273.h03v07.005.2007285103507.hdf" ;
>>>>>>> create title
>>>>>>> res@gsnLeftString = "Field name: " + data@long_name
>>>>>>> res@gsnRightString = "Units: " + data@units
>>>>>>>
>>>>>>> plot=gsn_csm_contour_map_ce(xwks,data,res) ; create plot
>>>>>>>
>>>>>>> ; cleaning up used resources
>>>>>>> delete(plot)
>>>>>>> delete(res)
>>>>>>> delete(xwks)
>>>>>>> delete(data)
>>>>>>> delete(data_raw)
>>>>>>> delete(eos_file)
>>>>>>> delete(he2_file)
>>>>>>>
>>>>>>> end
>>>>>>>
>>>>>>>
>>>>>>> The plotting elaboration is very slow, my PC seem not working;
>>>>>>>
>>>>>>> commenting the "zoom" part:
>>>>>>>
>>>>>>> ;res@mpMinLatF = min(data@lat2d)
>>>>>>> ;res@mpMaxLatF = max(data@lat2d)
>>>>>>> ;res@mpMinLonF = min(data@lon2d)
>>>>>>> ;res@mpMaxLonF = max(data@lon2d)
>>>>>>>
>>>>>>> plotting elaboration time is about 2 minutes.
>>>>>>>
>>>>>>> Are there errors in the script or my computer is a clunker?
>>>>>>>
>>>>>>> Thank you so much
>>>>>>>
>>>>>>> Filippo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ncl-talk mailing list
>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>
>>>>>> _______________________________________________
>>>>>> ncl-talk mailing list
>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
Received on Thu Dec 27 13:05:37 2012