Bug #1078

Bug in gmt_nc.c for grids using 64-bit addressing

Added by Paul 18 days ago. Updated 9 days ago.

Status:FeedbackStart date:2017-04-08
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:Candidate for next bugfix release
Affected version:5.x-svn Platform:

Description

This is a clarification of issue #1019. The key problem is a bug in gmt_nc.c which only appears when the grid exceeds 32-bit addressing. The following example illustrates the problem:

# NC4 format using byte output
gmt grdlandmask -r -Rd -Dl -I10s -N0/1 -Gnc4.nc=nb
gmt grdimage nc4_chunk.nc=nb -Jx0.0001i -P -Ct.cpt -Anc4.png
# Native C using byte C output
gmt grdlandmask -r -Rd -Dl -I10s -N0/1 -Gbyte.grd=bb
gmt grdimage byte.grd=bb -Jx0.0001i -P -Ct.cpt -Abyte.png

Both commands produce a humongous 129600 by 64800 grid for a total of 7.8 Gb nodes, far beyond the 32-bit 4 Gb limit. To run this, grdimage needs to allocate almost 60 Gb of array space. However, GMT uses 64-bit indexing throughout. Attached are the two png plots produced by grdimage (which uses GDAL to write the PNG images). As you see, the PNG produced by the native byte.grd grid is correct (hence grdlandmask is handling the large grid correctly) while the same command, when written via netCDF, shows typical symptoms of 32/64-bit snafu. I have looked long and hard (and made some changes) to gmt_nc.c but the problem remains. Note: I am assuming the problem lies in GMT since netCDF is used by so many groups for very large data that it is unlikely there is a limitation in netCDF; however, this is an assumption to consider as well.

Screen Shot 2017-04-08 at 4.09.28 PM.png - PNG produced via netCDF grid (430 KB) Paul, 2017-04-08 17:01

Screen Shot 2017-04-08 at 4.52.17 PM.png - PNG produced via native grid (303 KB) Paul, 2017-04-08 17:02

grab.jpeg (220 KB) Paul, 2017-04-17 10:14

History

#1 Updated by Remko 11 days ago

I do not see the error either. Do not forget that the bug might be in reading rather than writing.
Have you tried to open nc4_chunk.nc with Panoply?
Also, I do not want to rule out a NetCDF bug here ... it could be in the way NetCDF does chunking.
Have you tried it with chunking switched off?

#2 Updated by Paul 11 days ago

I have done this with no chunking too, no difference. Will try Panoply after Easter.

#3 Updated by Paul 10 days ago

  • File grab.jpeg added
  • Status changed from New to Feedback

This is what the nc4_chunk.nc looks like in Panoply. No chuncking (i.e., classic) makes no difference. So it definitively is in the writing. I also wrote directly via GDAL as netcdf and it looks good. So it has to be our gmt_nc.c code doing something 32-bitish.

#4 Updated by Remko 9 days ago

Thanks for the feedback. That makes it easier to trace.

Also available in: Atom PDF