Starting in the summer of 2012, ASF undertook the significant challenge of developing a Seasat telemetry decoder in order to create raw data files suitable for focusing by a synthetic aperture radar (SAR) correlator. In this case, that means processible by ROI, the Repeat Orbit Interferometry package developed at Jet Propulsion Laboratory. In addition to creating the range lines out of minor frames, the decoder must interpret the 18 fields in the headers to create a metadata file describing the state of the satellite when the data was collected.
The main challenges in decoding the raw telemetry were:
- Overcoming bit error problems
- Properly forming major lines from a variable number of minor frames
- Maintaining sync lock
- Discovering sentinels marking data collection boundaries
2.1 Problems with Bit Fields
All 10 of the bit fields proved to be unreliable, and, thus, with the exception of the fill flag, they are ignored by all of the software developed during this project. This section describes the ways in which the bit fields are unreliable.
Each Seasat minor frame contains 8 bits to record time and status. These bits encode 18 metadata fields, subcommutated in the first 10 minor frames of each range line. There are 10 bit fields, four fields that should be constant for a data take, two fields that should change rarely and two fields that should change steadily. Unfortunately, due to the high bit error rate (BER) of the telemetry data, even fields that should be constant show high variability. The following plots, showing decoded metadata values plotted over 800,000 range lines, drive home the enormity of the bit error problems in these data.
Bits per Sample
Bits Per Sample: This value should always be 5, as the parameter never changed during the entire Seasat mission.
PRF rate code
PRF Rate Code: The PRF rate code should be a 4 for the entire mission, since this satellite parameter never changed.
Last Digit of Year
Last Digit of Year: The Seasat mission occurred entirely in 1978, so the last digit of the year should always be 8.
Station Code
Station Code: The Station Code should be a constant for any given datatake. 5: Fairbanks, Alaska; 6: Goldstone, Calif.; 7: Merrit Island, Fla.; 9: Oak Hanger, United Kingdom; 10: Shoe Cove, Newfoundland.
Day of Year
Day of Year: For a given datatake, the day of year should change at most once, since any single datatake cannot exceed even an hour in duration, much less an entire day.
Delay to Digitization
Delay to Digitization: The delay to digitization parameterizes the time between emission of a pulse from the satellite and recording of return echoes. Used to calculate the slant range to the first pixel, the delay should change only a handful of times in any given raw data file based upon changes in orbital altitude.
Clock Drift
Clock Drift: The spacecraft clock drift records the timing error of the spacecraft clock. This should be a smoothly changing field, generally in the 2,000- to 3,000-millisecond range. It is not known how this field was originally created, only that it is vital in getting reasonable geolocations for processed imagery.
MSEC
MSEC: This field records the millisecond of the day when the data were acquired and should, therefore, be a linearly increasing field with an exact slope of 1/PRF.
Given all of the fallout from these truly bizarre plots, it is no surprise that attempts to use the fill flag quickly proved difficult; the bit errors are so pervasive that the field is unreliable.
2.2 Determining Minor Frame Numbers
Several oddities in the raw data are exacerbated by the high BER. First, the data are organized into 1,180-bit minor frames. This means that they are each 147.5 bytes long. Although the .5 byte offset was easy to deal with, it turns out that sync codes may actually appear at 147, 147.5, or 148 bytes from each other at seemingly random places in the raw data file – a topic addressed in section 2.3
Moreover, a variable number of minor frames need to be combined to create a single range line. Some lines contain 59 minor frames and some contain 60. Considering that the frame number in the minor frames is only 7 bits, and no major frame numbers exist in the telemetry, the “simple” task of finding the start of each major line was at times quite difficult. Synchronization codes can be either byte-aligned or non-byte-aligned, and partial lines occur on a regular basis. As a result, the minor frame numbers eventually had to be determined by context.
The current frame number is determined using three previous minor frame numbers and the next frame number — along with a handful of heuristics. For example, if a gap is found in consecutive minor frame numbers, the following rules are applied:
- If the next frame is 1 and last was either 59 or 60, assume this is frame zero
- Else if (next_frame-last_frame)==2, put this frame in sequence
- Else if last two frames are in sequence, try to put this one in sequence – If the last frame < 59, put this in sequence – If the last frame was 59 and the last line was length 60, then this HAS to be frame zero since we never have two length 60 lines in a row.
- Else if last2 and last3 frames are in sequence and last frame is 0, then set this frame to 1
- If we got to here, then we did not fix the error!
Even beyond these rules, additional checks for a bad frame number 0 and major frames that spuriously showed more than 60 minor frames still had to be performed.
Aside: Bit Errors in Frame Numbers
Random bit errors in the middle of a line: In these decoded minor frame numbers, we see that lines 1, 2, 4 and 5 have no frame number errors; they are in sequence starting from 1 and going up to 59 or 60 by ones. Meanwhile, line 3 has 24 frame numbers in a row that are in error.
Partial lines: The first line is missing minor frames 5-9; the second line is complete; the third line has repeated minor frames 15-19; and the fourth line is the decoder’s attempt to enforce the fact that at most 60 minor frames form a range line. The fifth line is missing minor frames 23-26; the sixth line is complete; the seventh line has repeated minor frames 32-36; and, again, the eighth line is the decoder’s attempt to enforce the fact that at most 60 minor frames form a range line.
Multiple lines of bit errors: This example shows how bad random bit errors can be, even if no minor frames are actually missing. Incredibly, in five lines, 122 minor frames are in error out of 298 total, giving a 40 percent error rate for these frame numbers. Perhaps even more incredibly, the ASF Seasat decoder managed to fix all of these frame numbers.
- Original non-fixed frame numbers:
- Frame numbers fixed by the ASF Seasat decoder:
2.3 Maintaining Sync Lock
One very important aspect of decoding telemetry data is maintaining a sync lock: The decoding program must be able to find the synchronization codes that occur at the beginning of each minor frame.
Early in the development of the decoder, it was determined that the sync codes are just as susceptible to bit errors as the rest of the data. Initially, finding sync codes required a considerable amount of searching in the file, with the hope that no false positives would be encountered. After much development and testing, it was determined that in order to maintain sync lock, some number of bit errors had to be allowed in the sync code. Therefore, the code was configured to allow 7 bit errors per sync code out of 24 bits. Values less than this needlessly split datatakes (single passes of data over a given ground station). Values greater than this showed too many “false positive” matches for sync codes.
As a result of this extensive analysis, a pattern was determined in the location of sync codes. That is, a byte-aligned, 147.5-byte frame followed by a non-byte-aligned, 147.5-byte frame, repeated 14,217 times, followed by a single instance of a 147-byte frame. In code form:
Once this pattern was established, most problems with locating sync codes were abated.
2.4 Data Sentinel Values – Breaking Datatakes
The next problem involved bad sections of data that defied attempts to match frame numbers. The only solution is to break the datatake into multiple pieces, closing the current output file when problems arise, and creating a new output file when sync is regained. This is much like what SyncPrep does, except that the ASF decoder has to be more stringent in its rules for maintaining sync since it must be able to properly build range lines in addition to just finding sync codes.
In addition to losing sync lock as a result of BER, two additional cases arose that will break a datatake into segments: either 60 occurrences of the fill flag in a row, or the repeated occurrence of frame number 127. The fill flag is a valid field but is so unreliable it can only be trusted to be correct after many consecutive hits. The frame number 127 showed to be a sentinel for no data; it occurred thousands of times in areas where no valid SAR data was being collected. Either of these happenings will also cause the ASF Seasat decoder to close the current output file and create a new one.
2.5 Results of Decoding
seasat_decoder:
- Decode raw signal data into unpackaged byte signal data (.dat file)
- 13680 unsigned bytes of signal data per line
- File size is aways lines * 13680 bytes in length
- Decodes all headers to ASCII (.hdr file)
- 20 columns of integer numbers per line
- One line entry per line of decoded signal data
- Additional Features:
- Allows both byte aligned and anon-byte aligned minor frames
- Deals with variable length lines, partial lines
- Fixes frame numbers from context if possible
- Creates one or more output files per input based on sentinels
- Assembles headers spread across 10 minor frames
In spite of all of the challenges and problems in the raw data, the ASF Seasat decoder is able to decode raw telemetry SAR data. Using five frame numbers in sequence and a handful of heuristics, telemetry data is decoded into byte-aligned, 8-bit samples. Concurrently, all of the metadata stored in the headers is decoded and placed in an external file.
The current strategy tried to err on the side of only allowing valid SAR data to be decoded. Still, 7-bit errors had to be allowed in a sync code match to even get through the raw data. In addition, the decoded header information is simply not reliable. For example, early in development, the ASF Seasat decoder broke one 7-GB chunk of raw data into 24 segments of decoded data, dumping a header at the beginning and ending of each segment. Analysis of the decoded times in these headers showed that of the 48 dumped, 3 were completely zero and an additional 12 were in error. In other words, the decoded times did not make sense in context with the surrounding time values.
Thus, even after completing the decoder development with bit error tolerance, frame number heuristics, proper sync code detection, and known sentinel values for good data boundaries, the decoded Seasat archives were still nearly unusable in any reliable fashion.
| Processing Stage | #Files | Size (GB) |
|---|---|---|
| Capture | 38 | 2610 |
| SyncPrep | 1840 | 2431 |
| Original Decoded | 1470 | 3585 |
Initial Data Recovery: 93 percent of the data captured from tape made it through SyncPrep; Approximately 92 percent of that data was decoded (assuming a 1.6-expansion factor).
Aside: ASF Tape Archive File Names
When the tapes were captured onto disk, files were named based upon tape number and section of tape read. For example, the first part of tape1 was initially named SEASAT_tape1_01Kto287K.
This file was run through SyncPrep, which created multiple subfiles based upon its ability to maintain a sync lock, sometimes creating over 100 such numbered files, e.g. SEASAT_tape1_01Kto287K.000 to SEASAT_tape1_280Kto668K.020
Next, the files go through the ASF Seasat decoder, gaining yet another subfile number, but the prefix “SEASAT_” is removed. Note that this stage creates a file pair of {.dat, .hdr}, e.g. tape1_01Kto287K.018_000, tape1_01Kto287K.018_001, and tape1_01Kto287K.018_002 file pairs were all created from a single decode of SEASAT_tape1_280Kto668K.018.
Thus, for a single captured file, SyncPrep could make tens to a few hundred data segments, while the ASF Seasat decoder could break each of these files into even more sub-segments.
Written by Tom Logan, July 2013
