GMT 5.1.1 grdreformat & grdcut - failure to allocate enough memory

Added by Charlotte 8 months ago

I have an ~3 GB GeoTiff (http://stream.princeton.edu/POLARIS/PROPERTIES/3arcsec/alpha_max_0_5.tif) that covers the lower 48 states at 3arcsecond resolution. I am trying to subset this file to my area of interest and save it as a GMT netcdf 32 Float file, to use in later processing with grdmath.

I keep getting gdalread errors regarding failure to allocate enough memory, see below for commands and corresponding errors. I do not get these errors for smaller files of the same extent that are 30arcsec (http://stream.princeton.edu/POLARIS/PROPERTIES/30arcsec/alpha_max_0_5.tif).

I am trying to figure out if I am using the best commands for the job: grdreformat & grdcut, or if there are other alternatives that would not give me the same error. As I am using university computers, I do not have administrative rights to update to a newer versions of GMT. Any insights are greatly appreciated. Thank you. - C

grdreformat alpha_max_0_5.tif -R-100/-80/36/49 test.grd -V

grdreformat: Translating file ./Test3.tif (format gd = Import/export through GDAL)
to file out3.grd (format nf = GMT netCDF format (32-bit float), COARDS, CF-1.5)
grdreformat: gdalread: failure to allocate enough memory
grdreformat: ERROR reading file with gdalread.
grdreformat (GMTAPI_Import_Grid): Could not create file [./Test3.tif=gd]
[Session gmt (0)]: Error returned from GMT API: GMT_GRID_READ_ERROR (18)
grdreformat: Error for input file: No such file (out3.grd)
grdreformat: Syntax error: Specify only one input and one output file
ERROR: Caught signal number 11 (Segmentation fault) at
/lib64/libc.so.6[0x32ec3343f2]
[0x0]
Stack backtrace:
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(sig_handler+0x13c)[0x2afde20a4adc]
/lib64/libpthread.so.0[0x32ece0f7e0]
/lib64/libc.so.6[0x32ec3343f2]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_grd_get_format+0x43)[0x2afde20d9bc3]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_grdreformat+0x2b5)[0x2afde2251cb5]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_Call_Module+0x151)[0x2afde20a8751]
gmt(main+0x26f)[0x400fcf]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x32ec21ed1d]
gmt[0x400c29]

grdcut alpha_max_0_5.tif -Gtest.grd -R-100/-80/36/49 -V

ERROR: Caught signal number 11 (Segmentation fault) at
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_gdal_read_grd+0x240)[0x2ae1b7cc7780]
[0x2ae22370c000]
Stack backtrace:
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(sig_handler+0x13c)[0x2ae1b7caaadc]
/lib64/libpthread.so.0[0x32ece0f7e0]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_gdal_read_grd+0x240)[0x2ae1b7cc7780]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_read_grd+0xa5)[0x2ae1b7ce01c5]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMTAPI_Import_Grid+0x1f7)[0x2ae1b7cb7567]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMTAPI_Import_Data+0x110)[0x2ae1b7cb9d20]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_Get_Data+0x8e)[0x2ae1b7cb9e0e]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_Read_Data+0xe1)[0x2ae1b7cb9f91]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_grdcut+0x9eb)[0x2ae1b7e1e43b]
/apps/rhel6/gmt/5.1.1/bin/../lib64/libgmt.so.5(GMT_Call_Module+0x151)[0x2ae1b7cae751]
grdcut(main+0x26f)[0x400fcf]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x32ec21ed1d]
grdcut[0x400c29]


Replies (4)

RE: GMT 5.1.1 grdreformat & grdcut - failure to allocate enough memory - Added by Joaquim 8 months ago

That area at 3 arc sec is still about 1.4 Gb of data. Maybe you simply do not have a contiguous chunk of memory available. Try also with GDAL to see if it's more efficient with memory consumption.

RE: GMT 5.1.1 grdreformat & grdcut - failure to allocate enough memory - Added by Charlotte 8 months ago

Thanks Joaquim! I get the same results from both the reformat and cut commands even when I adjust the extent to be smaller (0.5 degrees in both latitude and longitude). I'm a little confused because I have processed larger files with my computer before. I tried to use gdal_translate -of netCDF -co "FORMAT=NC4" .tif .nc , but got the resulting error:

gdal_translate: error while loading shared libraries: libnetcdf.so.6: cannot open shared object file: No such file or directory

gridinfo:
Title: Grid imported via GDAL
Command:
Remark:
Gridline node registration used [Cartesian grid]
Grid file format: gd = Import/export through GDAL
x_min: -124.999583333 x_max: -66.0004166667 x_inc: 0.000833333333333 name: x nx: 70800
y_min: 23.0004166667 y_max: 49.9995833333 y_inc: 0.000833333333333 name: y ny: 32400
z_min: 0.177513569593 z_max: 77.1769714355 name: z
scale_factor: 1 add_offset: 0

gdalinfo:
gdalinfo: error while loading shared libraries: libnetcdf.so.6: cannot open shared object file: No such file or directory

RE: GMT 5.1.1 grdreformat & grdcut - failure to allocate enough memory - Added by Paul 8 months ago

Time to tell your sys admin they need to update their distros since libraries are getting out of date. Also GMT 5.1.1 was released, hm, a while a go (4 years is an eternity in GMT development).

RE: GMT 5.1.1 grdreformat & grdcut - failure to allocate enough memory - Added by Charlotte 8 months ago

An updated version of gdal was installed and I was able to use gdal_translate to first convert the .tif to .nc, then GMT 5.1.1 had no problems with grdreformat and grdcut on that .nc file! Thanks again for the suggestions, maybe I can get the higher powers to update GMT soon!

(1-4/4)