Bug #1036

GRDTRACK (and related GDAL issue)

Added by Mike 5 months ago. Updated 2 months ago.

Status:FeedbackStart date:2017-01-27
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Affected version:5.2.x Platform:

Description

I have a very very big cartesian grid (23 Gb)
If I try to read it in to gmt with gdal driver Gtiff in preparation for grdtrack I get the following:
grdtrack: Processing input grid(s)
grdtrack: Object ID 0 : Registered GMT Grid File /data/Work/ARCTICDEM/LOWREZ_DEMS/CANADA/cded_250k_mos_POLAR.tif as an Input resource with geometry Surface [n_objects = 1]
grdtrack: GMTAPI_Begin_IO: Input resource access is now enabled [container]
grdtrack: GMTAPI_Import_Grid: Passed ID = 0 and mode = 1
grdtrack: File /data/Work/ARCTICDEM/LOWREZ_DEMS/CANADA/cded_250k_mos_POLAR.tif reads with GDAL driver GTiff
grdtrack: Grid is Cartesian
grdtrack: GMT_End_IO: Input resource access is now disabled
grdtrack: GMTAPI_Begin_IO: Input resource access is now enabled [container]
grdtrack: GMTAPI_Import_Grid: Passed ID = 0 and mode = 2
grdtrack: Reading grid from file /data/Work/ARCTICDEM/LOWREZ_DEMS/CANADA/cded_250k_mos_POLAR.tif
grdtrack: gdalread: failure to allocate enough memory
grdtrack: ERROR reading file with gdalread.
grdtrack (GMTAPI_Import_Grid): Could not open file [/data/Work/ARCTICDEM/LOWREZ_DEMS/CANADA/cded_250k_mos_POLAR.tif]
[Session gmt (0)]: Error returned from GMT API: GMT_GRID_READ_ERROR (18)
[Session gmt (0)]: Error returned from GMT API: GMT_GRID_READ_ERROR (18)
[Session gmt (0)]: Error returned from GMT API: GMT_GRID_READ_ERROR (18)
grdtrack: GMT_Garbage_Collection: Destroying object: C=0 A=0 ID=0 W=Input F=GMT Grid M=File S=Unused P=0 D=5562e6f88a10 N=/data/Work/ARCTICDEM/LOWREZ_DEMS/CANADA/cded_250k_mos_POLAR.tif
grdtrack: GMTAPI_Garbage_Collection freed 1 memory objects
grdtrack: GMTAPI_Unregister_IO: Unregistering object no 0 [n_objects = 0]
gmt: Entering GMT_Destroy_Session

I have 256Gb of memory on this machine, so this seems unlikely - I have also managed to ingest files this large this way before.

Anyway as a work around I convert the file to netCDF with gdal (gdal_translate -of netCDF)
and throw grdtrack at it again. This time everything works out and I get a text file back with my million or so sampling points on it.
However there are tens of thousands of points returned which should be NaN, that have spurious z values in them. To clarify - the DEM I am sampling has NaN at those positions, but
grdtrack is producing z values.

Can someone explain why I would be getting these false positives?

Here is the code:

gmt grdtrack ${file}.csv -G$DEM -nn -Vd | sed 's/\t/ /g' | gawk '$0 !~ /NaN/' > output.txt

$DEM is the large DEM which looks fine in any number of gis packages. $file.csv is the input tracks I want to sample (~100,000 points, with the first two columns in the same projection as the DEM)

Thanks

History

#1 Updated by Paul 5 months ago

  • Status changed from New to Feedback

So likely a 32-bit variable in gmt_gdalread that is acting up. I know it is kind of big but if the data is not proprietary would it be possible to get access to the large TIFF file so I could run this in debug?
As for your grdtrack question: I don't know why of course, but we do want to find out. Is it possible to isolate a few of the points that should yield nan but yield garbage instead, i.e.,if you make an input file with just a few of those points, does it still return garbage. if so, please provide that input file as well.

#2 Updated by Mike 5 months ago

Hi Paul,

The error is indeed in the DEM - it had multiple different values for Not a Number in it.
Once these were stripped out it seemed to work ok. I can share the "bad" file with you, all the
same, if you feel it would be instructive.

Cheers

#3 Updated by Paul 2 months ago

Sorry dropping the ball on this. I would be interested in recreating the memory fault you reported, so if you could make me a file to ftp then I would like to try to find the 32-bit problem,

Also available in: Atom PDF