x2sys_cross time issues part II

Added by Bernard 3 months ago

Folks

I am having a version and developing an aversion of the problem reported by a graduate student here. I will be brief.

I set up the data base. All seems to go well, but when I attempt to run crossovers, I come to grief.

I am using UNIX time since the date of inception 01/01/70 00:00:00.000 in my .trk files.

Using the following .def file;

  1. Define file for the GRAV-D format
    #ASCII # File is ASCII
    #GEO # Data are geographic
    #MULTISEG #Data are multi-segment
    #name type NaN NaN-proxy scale offset oformat
    lat a N 0 1 0 %9.6f
    lon a N 0 1 0 %10.6f
    time a N 0 1 0 %14.3f
    grav a N 0 1 0 %6.2f
    magn a N 0 1 0 %7.2f
    bthy a N 0 1 0 %7.2f

A short section of one;

60.442626 204.675369 1215908734.000 33.92 NaN 315.62
60.441609 204.676953 1215908735.000 33.97 NaN 319.10
60.440593 204.678538 1215908736.000 34.03 NaN 323.01
60.439576 204.680122 1215908737.000 34.07 NaN 327.53
60.438560 204.681705 1215908738.000 34.12 NaN 333.00
60.437543 204.683288 1215908739.000 34.17 NaN 339.67
60.436527 204.684870 1215908740.000 34.22 NaN 347.59
60.435511 204.686453 1215908741.000 34.26 NaN 356.85
60.434496 204.688035 1215908742.000 34.31 NaN 367.24
60.433480 204.689616 1215908743.000 34.35 NaN 378.49
60.432464 204.691197 1215908744.000 34.39 NaN 390.53
60.431449 204.692778 1215908745.000 34.42 NaN 403.25
60.430433 204.694358 1215908746.000 34.47 NaN 415.86

So, it is as reported, latitude, longitude, time, gravity, a non-field and elevation. This is airborne data reduced
to the geoid.

The data as received were sequential in track line number, not in time, but I have used the UNIX command sort
to put them in order.

Attempting the internal crossovers. I get the following cascade of errors.

x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN01 - AN01 : x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
This continues to apparent infinity.

Regretfully each and every time value in the third column is greater than the one that precedes it. Usually by a second.

I can redo the .def file and name the time column something like "rtime" and it will run, but I am informed that x2sys_cross did not find time
in my files. It runs with some dummy times and does the crossover calculations, but does not produce velocities or real times for the events
of interest.

I am using 5.4.5_1. The I invoke x2sys_cross with "x2sys_cross $block -T$TAG -Ic -Qi -V -Z"

The block refers to the block of the airborne survey. It is something like AN0[1-9] or AS0[1-9]. The TAG is GRAVD.

Any and all assistance will be appreciated.

Bernie


Replies (13)

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

The error message comes from a spline interpolation where y(x) is passed in and it is found that x is not monotonically in/de-creasing. Very often this is actually means you have duplicate records with the same time. They may be sorted, but if you have two like this

60.442626 204.675369 1215908734.000 33.92 NaN 315.62
60.441609 204.676953 1215908735.000 33.97 NaN 319.10
60.440593 204.678538 1215908735.000 34.03 NaN 323.01

then it will complain. Less likely is that x is truly not monotonic and that would require a debug I think.

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

Paul

Thanks for this. I checked the files. They are monotonic without duplicate or reverse time entries.

I’ve decimated the data down to 10 seconds (0.1 Hz). One second data was kind of overkill, even for airborne data. This seems to help.

Now it runs quite quickly, finding mostly reasonable numbers of crossovers;

COA-calculate GRAVD
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN01 - AN01 : x2sys_cross: 921
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN02 - AN02 : x2sys_cross: 31
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN03 - AN03 : x2sys_cross: 92
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN04 - AN04 : x2sys_cross: 51
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN05 - AN05 : x2sys_cross: 67911
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN06 - AN06 : x2sys_cross: 178
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN07 - AN07 : x2sys_cross: 414
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN08 - AN08 : x2sys_cross: 529
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN09 - AN09 : x2sys_cross: 1810
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS01 - AS01 : x2sys_cross: 35493
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS02 - AS02 : x2sys_cross: 325
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS03 - AS03 : x2sys_cross: 141
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS04 - AS04 : x2sys_cross: 828
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS05 - AS05 : x2sys_cross: 636
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS06 - AS06 : x2sys_cross: 2268
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS07 - AS07 : x2sys_cross: 132
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS08 - AS08 : x2sys_cross: 20729
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS09 - AS09 : x2sys_cross: 427

The new weirdness is that it does not actually write out the crossovers. I redirect stdout to a file. This is what’s inside;

  1. Command : x2sys_cross AN01 -TGRAVD -Ia -Qi -V -Z
  2. Command : x2sys_cross AN02 -TGRAVD -Ia -Qi -V -Z
  3. Command : x2sys_cross AN03 -TGRAVD -Ia -Qi -V -Z
  4. Command : x2sys_cross AN04 -TGRAVD -Ia -Qi -V -Z
  5. Command : x2sys_cross AN05 -TGRAVD -Ia -Qi -V -Z
  6. Command : x2sys_cross AN06 -TGRAVD -Ia -Qi -V -Z
  7. Command : x2sys_cross AN07 -TGRAVD -Ia -Qi -V -Z
  8. Command : x2sys_cross AN08 -TGRAVD -Ia -Qi -V -Z
  9. Command : x2sys_cross AN09 -TGRAVD -Ia -Qi -V -Z
  10. Command : x2sys_cross AS01 -TGRAVD -Ia -Qi -V -Z
  11. Command : x2sys_cross AS02 -TGRAVD -Ia -Qi -V -Z
  12. Command : x2sys_cross AS03 -TGRAVD -Ia -Qi -V -Z
  13. Command : x2sys_cross AS04 -TGRAVD -Ia -Qi -V -Z
  14. Command : x2sys_cross AS05 -TGRAVD -Ia -Qi -V -Z
  15. Command : x2sys_cross AS06 -TGRAVD -Ia -Qi -V -Z
  16. Command : x2sys_cross AS07 -TGRAVD -Ia -Qi -V -Z
  17. Command : x2sys_cross AS08 -TGRAVD -Ia -Qi -V -Z
  18. Command : x2sys_cross AS09 -TGRAVD -Ia -Qi -V -Z

That’s one line for each block. No more.

The current version of the init command is;

gmt x2sys_init $TAG -D$X2SYS_HOME/$TAG.def -Etrk -F -Gg -Wdt120 -V

The current version of the x2sys_cross command is list above.

I can bring back the original problem that inspired the first message by removing the -Wdt[whatever] from the initiation of the database.

So it appears to be running smoothly, but not putting out any crossover information. Thoughts?

Thanks

Bernie

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

Sorry, all I can do is ask you to make me the tiniest file and example commands (starting with x2sys_init and def file) so I can run in debugger. Also, quite busy as Chair so may have to be a week until I have time to do the check.

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

Paul

I understand completely and appreciate any time to have to poke into this.

I have attached the track file for a single block, AN01. I have checked it. There are no repeat or reverse time records in the file.

The command I use to initialize the database is;

gmt x2sys_init $TAG -D$X2SYS_HOME/$TAG.def -Etrk -F -Gg -Ndk -Nse -Wdt120 -V

The entire script I use, including calls to x2sys_binlist and x2sys_put is attached as Init-Database.

The text of the script I use to calculate internal crossovers;

#!/bin/csh

setenv X2SYS_HOME `pwd`

set TAG = $1

gmt set FORMAT_FLOAT_OUT %.7f TIME_EPOCH 1970-01-01T00:00:00 TIME_UNIT s

set blocks = "AN01 AN02 AN03 AN04 AN05 AN06 AN07 AN08 AN09 AS01 AS02 AS03 AS04 AS05 AS06 AS07 AS08 AS09"

if (!(-e $TAG/crossover-files)) then
mkdir $TAG/crossover-files
endif

set destination = $TAG/crossover-files/external
set destinationint = $TAG/crossover-files/internal

if (!(-e $destination)) then
mkdir $destination
endif

if (!(-e $destinationint)) then
mkdir $destinationint
endif

if (-e $destinationint/coa.int) then
/bin/rm $destinationint/coa.int
endif

foreach block (`echo $blocks`)
if (!(-e $destinationint/$block.coa.int)) then
x2sys_cross $block -T$TAG -Ia -Qi -V -Z >> $destinationint/coa.int
endif
end

Thanks again.

Bernie

GRAVD.def - GRAVD def file (332 Bytes)

AN01.trk - Track file for block AN01, sampled at 0.1 Hz (10.9 MB)

Init-Database - Script to initialize the database. (870 Bytes)

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

Oops. Attached the 1 Hz file for AN01. Here is the correct file.

AN01.trk (1.09 MB)

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

Problem was an error in your x2sys_init command compounded by a bad check that failed to detect the error. You had

-Wdt120

which is wrong (only d or t can be given). This let the distance to be set by trying to parse the distance 't' which gave 0. Then, our check failed to check for less than or equal to 0 and instead checked just for negative distances.

I have fixed this via a pull request. However, just fix your -W option and you should be good to go.

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

Okay. Fixed that. My bad. My command of the obvious is slipping.

I redid the crossovers. Database creation progressed normally.

When I attempted crossovers using this command;

x2sys_cross $block -T$TAG -Ia -Qi -V -Z >> $destinationint/coa.int

I got a semi-infinite list of the following.
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!
x2sys_cross: Error: x-values are not monotonically increasing/decreasing (at zero-based record 2)!

I know the files are sequential without redundant time tags. I checked.

I changed to linear interpolation and that seemed to run normally with mostly reasonable results.
x2sys_cross $block -T$TAG -Il -Qi -V -Z >> $destinationint/coa.int

COA-calculate GRAVD
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN01 - AN01 : x2sys_cross: 921
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN02 - AN02 : x2sys_cross: 31
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN03 - AN03 : x2sys_cross: 93
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN04 - AN04 : x2sys_cross: 51
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN05 - AN05 : x2sys_cross: 264087
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN06 - AN06 : x2sys_cross: 178
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN07 - AN07 : x2sys_cross: 414
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN08 - AN08 : x2sys_cross: 529
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AN09 - AN09 : x2sys_cross: 2756
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS01 - AS01 : x2sys_cross: 133945
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS02 - AS02 : x2sys_cross: 325
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS03 - AS03 : x2sys_cross: 141
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS04 - AS04 : x2sys_cross: 821
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS05 - AS05 : x2sys_cross: 636
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS06 - AS06 : x2sys_cross: 5713
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS07 - AS07 : x2sys_cross: 132
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS08 - AS08 : x2sys_cross: 74411
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: Processing AS09 - AS09 : x2sys_cross: 427

Looking into the crossover file;

  1. Tag: GRAVD
  2. Command: x2sys_cross AN01 -TGRAVD -Il -Qi -V -Z
  3. lon lat t_1 t_2 dist_1 dist_2 head_1 head_2 vel_1 vel_2 grav_1 grav_2 magn_1 magn_2 bthy_1 bthy_2

AN01 0 AN01 0 12477-01-01T00:00:00/12797-01-01T00:00:00/28855.1 12477-01-01T00:00:00/12797-01-01T00:00:00/28855.1

-154.4167203 63.5044635 12485-01-01T00:00:00 12494-01-01T00:00:00 9704.3683896 23235.7517594 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.4302010 63.5178902 12485-01-01T00:00:00 12477-01-01T00:00:00 9706.0016098 624.5543961 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.4967028 63.5841254 12485-01-01T00:00:00 12477-01-01T00:00:00 9714.0584643 269.1955565 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.5618806 63.6490419 12485-01-01T00:00:00 12478-01-01T00:00:00 9721.9549122 2641.2435650 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.6258840 63.7127889 12485-01-01T00:00:00 12478-01-01T00:00:00 9729.7090922 2988.6730710 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.6930572 63.7796929 12485-01-01T00:00:00 12478-01-01T00:00:00 9737.8472912 3571.3033118 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.7554985 63.8418839 12485-01-01T00:00:00 12478-01-01T00:00:00 9745.4122118 3891.8959805 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.8230746 63.9091891 12485-01-01T00:00:00 12479-01-01T00:00:00 9753.5992128 4497.4089570 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.8852005 63.9710661 12485-01-01T00:00:00 12479-01-01T00:00:00 9761.1259302 4822.7814367 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-154.9530935 64.0386869 12485-01-01T00:00:00 12483-01-01T00:00:00 9769.3513279 5447.8573286 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
-155.0149466 64.1002922 12485-01-01T00:00:00 12483-01-01T00:00:00 9776.8449864 5767.6971242 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

This is just part of the output.

In the good news department, it is attempting to interpret the time field. However insensibly. Nothing is coming out but NaN for everything
to the right of the 2nd distance along track. The distance along track seems implausible in the default units (km).

Thoughts?

Thanks again for your time.

Bernie

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

The times are messed up because your format file says column 3 is time (i.e., absolute time) but in fact it should be rtime (relative time). You times are all relative to the epoch. From the x2sys_iniit documentation:

name is the name of the column variable. You must use the special names lon (or x if Cartesian) and lat (or y) for the two required coordinate columns, time when optional absolute time data are present, and rtime when relative time data are given (make sure the GMT defaults TIME_UNIT and TIME_EPOCH are set properly). Regardless of input time flavor, we will write absolute time on output.

With time the crossover records look more hopeful:

-149.2652358 63.6477656 2009-07-16T19:39:47 2010-07-20T22:03:36 880.4437244 27621.6986249 329.4059669 57.4308089 116.2512367 131.8343995 70.2434670 68.5999095 NaN NaN 1101.0365893 1095.0502580

FYI, all bug reporting should take place at our GitHub site: https://github.com/GenericMappingTools/gmt since GMT was transition from subversion to git last year.

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

I will put the next one on Github. Sorry for my oversight.

I have tried rtime before. This results in the following output;

COA-calculate GRAVD
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: No time column, use dummy times
x2sys_cross: Processing AN01 - AN01 : x2sys_cross: 467
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: No time column, use dummy times
x2sys_cross: Processing AN02 - AN02 : x2sys_cross: 0
x2sys_cross: Files found: 1
x2sys_cross: Checking for duplicates : x2sys_cross: 0 found
x2sys_cross: No time column, use dummy times
x2sys_cross: Processing AN03 - AN03 : x2sys_cross: 34

And in the resultant crossover file;
Command : x2sys_cross AN01 -TGRAVD -Il -Qi -V -Z
  1. Tag: GRAVD
  2. Command: x2sys_cross AN01 -TGRAVD -Il -Qi -V -Z
  3. lon lat i_1 i_2 dist_1 dist_2 head_1 head_2 vel_1 vel_2 rtime_1 rtime_2 grav_1 grav_2 magn_1 magn_2 bthy_1 bthy_2

AN01 0 AN01 0 NaN/NaN/28855.1 NaN/NaN/28855.1

-155.7041743 63.4568640 14866.7807351 977.6787995 10162.2702020 560.7285485 352.0367557 263.3260198 NaN NaN 1248569620.9036756 1247770395.3939974 19.9046515 20.2275160 NaN NaN 493.2479718 494.6717871
-154.8995780 63.4968127 33975.6352371 1047.7489811 23288.7272546 601.0707616 173.2822058 264.1286929 NaN NaN 1249494902.1761854 1247770745.7449057 29.0866002 28.5427140 NaN NaN 680.9156789 684.2247258
-155.7251361 63.5253102 14851.0799282 584.5242549 10154.5677164 330.5964794 352.3438883 83.6262795 NaN NaN 1248569542.3996408 1247767570.6212745 16.2959856 17.9796084 NaN NaN 414.6553841 413.1791946
-154.1010411 63.5320227 14312.4920773 1117.1538949 9603.1705412 641.0056101 353.7688478 264.5621046 NaN NaN 1248548390.4603868 1247771092.7694745 42.5172976 40.0853451 NaN NaN 643.9975678 646.8666050

Note that the velocity is not calculated. I suppose that is because the program does not believe the time in the .trk files.

Just FYI, I am setting the TIME_UNIT and TIME_EPOCH;

gmt set TIME_EPOCH 1970-01-01T00:00:00 TIME_UNIT s

I am using version 5.4.5 of x2sys_cross.

Sorry to be a pest about this, but I am trying to make it work and work correctly.

Thanks

Bernie

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

Would you be able to try with GMT 6.x in GibHub? Clearly it seems to work for me. it is much harder and more work for us to devote time to fix issues in 5.4.x as we are working to release 6.x

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

I understand. No problem. I will try 6.x.

Do you have an idea when you will do an official release of GMT 6?

RE: x2sys_cross time issues part II - Added by Paul 3 months ago

We hope it will happened during the summer.

RE: x2sys_cross time issues part II - Added by Bernard 3 months ago

Version 6.whatever works.

Thanks for your help.

(1-13/13)