ch10_handbook:data_file_interpretation
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
ch10_handbook:data_file_interpretation [2014/04/13 12:09] – bob | ch10_handbook:data_file_interpretation [2014/07/16 15:18] (current) – Added links to TSPI and CAN Bus pages bob | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ~~ODT~~ | ||
===== DATA FILE INTERPRETATION ===== | ===== DATA FILE INTERPRETATION ===== | ||
Line 71: | Line 72: | ||
==== Overall Data Packet Organization ==== | ==== Overall Data Packet Organization ==== | ||
- | Overall data packet organization is shown in Figure 6-1. Data packets contain a standard | + | Overall data packet organization is shown below. Data packets contain a standard |
header, a data payload containing one or multiple data messages, and a standard trailer. The | header, a data payload containing one or multiple data messages, and a standard trailer. The | ||
standard header is composed of a required header, optionally followed by a secondary header. | standard header is composed of a required header, optionally followed by a secondary header. | ||
Line 100: | Line 101: | ||
| DATA CHECKSUM | **Packet Trailer** | | | DATA CHECKSUM | **Packet Trailer** | | ||
- | **Data Packet Organization** | + | < |
Data packets must contain data. They are not allowed to only contain filler. Filler can be | Data packets must contain data. They are not allowed to only contain filler. Filler can be | ||
Line 136: | Line 137: | ||
The packet header contains information about the data payload such as time, packet | The packet header contains information about the data payload such as time, packet | ||
length, data type, data version, and other information. The layout of a Chapter 10 packet header | length, data type, data version, and other information. The layout of a Chapter 10 packet header | ||
- | is shown in Figure 6-2. FIXME | + | is shown below. |
< | < | ||
struct SuI106Ch10Header | struct SuI106Ch10Header | ||
{ | { | ||
- | uint16_t uSync; | + | uint16_t |
- | uint16_t uChID; | + | uint16_t |
- | uint32_t ulPacketLen; | + | uint32_t |
- | uint32_t ulDataLen; | + | uint32_t |
- | uint8_t ubyDataVer; | + | uint8_t |
- | uint8_t ubySeqNum; | + | uint8_t |
- | uint8_t ubyPacketFlags; | + | uint8_t |
- | uint8_t ubyDataType; | + | uint8_t |
- | uint8_t aubyRelTime[6]; | + | uint8_t |
- | uint16_t uChecksum; | + | uint16_t |
}; | }; | ||
</ | </ | ||
- | **Packet header structure.** | + | |
+ | < | ||
The Channel ID field uniquely identifies the source of the data. The value of the | The Channel ID field uniquely identifies the source of the data. The value of the | ||
Line 197: | Line 199: | ||
The optional secondary header is used to provide an absolute time (i.e. clock time) stamp | The optional secondary header is used to provide an absolute time (i.e. clock time) stamp | ||
for data packets. The secondary header time format can be interpreted several ways. The | for data packets. The secondary header time format can be interpreted several ways. The | ||
- | specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure | + | specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure |
- | Figure 6-3 is used when secondary header time is to be interpreted as a Chapter 4 format value | + | is used when secondary header time is to be interpreted as a Chapter 4 format value |
- | (Flag Bits 3-2 = 0). The structure | + | (Flag Bits 3-2 = 0). The following |
interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1). | interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1). | ||
Line 213: | Line 215: | ||
}; | }; | ||
</ | </ | ||
- | **Optional secondary header structure with IRIG 106 Ch 4 time representation.** | + | < |
- | < | + | < |
struct SuI106Ch10SecHeader_1588Time | struct SuI106Ch10SecHeader_1588Time | ||
{ | { | ||
Line 224: | Line 226: | ||
}; | }; | ||
</ | </ | ||
- | **Optional secondary header structure with IEEE 1588 time representation.** | + | < |
==== Data Payload ==== | ==== Data Payload ==== | ||
Line 240: | Line 242: | ||
when there are no more data messages to read. | when there are no more data messages to read. | ||
- | Intra-packet headers, when they are present, typically contain one, or sometimes more | + | Intra-packet headers, when they are present, typically contain one, or sometimes more |
than one, time stamp as well as other information about the data message that follows. | than one, time stamp as well as other information about the data message that follows. | ||
- | Commonly used structures for intra-packet time data are shown in Figure 6-5, Figure 6-6, and | + | Commonly used structures for intra-packet time data are shown in the three figures below. |
- | Figure 6-7. These three time structures will be referenced in most of the data format descriptions | + | These three time structures will be referenced in most of the data format descriptions |
that follow. | that follow. | ||
Line 253: | Line 255: | ||
}; | }; | ||
</ | </ | ||
- | **Intra-packet Time Stamp, 48-bit RTC.** | + | < |
< | < | ||
Line 264: | Line 266: | ||
}; | }; | ||
</ | </ | ||
- | **Intra-packet Time Stamp, IRIG 106 Ch 4 binary** | + | < |
< | < | ||
Line 273: | Line 275: | ||
}; | }; | ||
</ | </ | ||
- | **Intra-packet Time Stamp, IEEE 1588** | + | < |
- | Computer Generated Data | + | ==== Data Formats ==== |
+ | |||
+ | [[Computer Generated Data]] | ||
+ | |||
+ | [[PCM Data]] | ||
+ | |||
+ | [[Time Data]] | ||
+ | |||
+ | [[MIL-STD-1553 Data]] | ||
+ | |||
+ | [[Analog Data]] | ||
+ | |||
+ | [[Discrete Data]] | ||
+ | |||
+ | [[Message Data]] | ||
+ | |||
+ | [[ARINC 429 Data]] | ||
+ | |||
+ | [[Video Data]] | ||
+ | |||
+ | [[Image Data]] | ||
+ | |||
+ | [[UART Data]] | ||
+ | |||
+ | [[IEEE 1394 Data]] | ||
+ | |||
+ | [[Parallel Data]] | ||
+ | |||
+ | [[Ethernet Data]] | ||
+ | |||
+ | [[TSPI Data]] | ||
+ | |||
+ | [[CAN Bus Data]] | ||
+ | |||
+ | ==== Time Interpretation ==== | ||
+ | |||
+ | | ||
+ | message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100 | ||
+ | nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it | ||
+ | could be initialized to zero or some random number. Some recorder vendors will preset the RTC | ||
+ | to a value based on absolute clock time, but for interoperability reasons it is unwise to assume | ||
+ | the RTC value will be related to absolute clock time in any meaningful fashion. | ||
+ | Absolute clock time comes into a recorder and is recorded much like any other data source. In | ||
+ | fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to | ||
+ | record input time signals. Since Time Data, Format 1 packets contain both the absolute input | ||
+ | time value and the RTC clock value at the instant the absolute time was valid, these packets can | ||
+ | be used to relate RTC values to the input absolute time source. For example, if a time packet is | ||
+ | recorded with a RTC value of 1,000,000 and an absolute time value of 100: | ||
+ | the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to | ||
+ | be 100: | ||
+ | |||
+ | When multiple time channels are available, it is incumbent on the programmer or data | ||
+ | analyst to determine and select the best source of time for a particular data set. For example, | ||
+ | there may be separate time channels for time derived from IRIG B, Global Positioning System | ||
+ | (GPS), and an internal battery backed up clock. In this scenario, all of these time sources are | ||
+ | present in the data file as separate channels, each correlating the RTC to its own notion of clock | ||
+ | time. The software application may allow the user to select which source of time to use for a | ||
+ | given analysis. Alternatively, | ||
+ | which time channels are providing valid time. In general, each time source will provide a | ||
+ | slightly (or not so slightly) different clock time. It is usually most correct to select one time | ||
+ | channel only and to use this channel exclusively to correlate RTC time to absolute clock time for | ||
+ | all data packet types. | ||
+ | |||
+ | The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least | ||
+ | as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on | ||
+ | the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It | ||
+ | is tempting for an application to find a single time packet early in a recording and to use those | ||
+ | time values to subsequently derive clock time from relative time. It is better to use the clock and | ||
+ | relative time values from a time packet that occurs near the current data packet as the data file is | ||
+ | decoded since there is some drift in the RTC during a recording session. It also may be the case | ||
+ | that there is a jump in input clock time during a recording, such as when GPS locks for the first | ||
+ | time, or when an IRIG time source is reprogrammed. | ||
+ | |||
+ | Index and Event Records | ||
+ | Often times it is useful to make an in-memory version of the data file index. This allows | ||
+ | rapid access to recorded data packets based on time or the occurrence of events. A general | ||
+ | algorithm for reading all root and node index packets is as follows: | ||
+ | |||
+ | 1. If " | ||
+ | |||
+ | 2. Move read pointer to last packet of data file. Store file offset of this packet. | ||
+ | |||
+ | 3. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3) | ||
+ | then index does not exist. | ||
+ | |||
+ | 4. Get the index count from the CSDW. | ||
+ | |||
+ | 5. For each root index contained in the packet, | ||
+ | * Read the Node Index offset value | ||
+ | * Move the read pointer to the Node Index offset value | ||
+ | * Read the Node Index packet | ||
+ | * Get the node index count from the CSDW | ||
+ | * For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values. | ||
+ | |||
+ | 6. Read last root node index. If offset value is equal to current root node packet offset | ||
+ | (stored in Step 2) then done. | ||
+ | |||
+ | 7. Else the move read pointer to the next Root Index packet offset value | ||
+ | |||
+ | 8. Read the next Root Index packet. | ||
+ | |||
+ | 9. Go to Step 4. | ||
+ | |||
+ | ==== Data Streaming ==== | ||
+ | |||
+ | | ||
+ | ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is | ||
+ | normally done over an Ethernet port, but any network connection that supports UDP/IP can use | ||
+ | this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the | ||
+ | use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more | ||
+ | specific unicast and multicast IP addresses, or broadcast address. Different channels can be | ||
+ | addressed to different addresses. | ||
+ | |||
+ | It is common to publish different groups of data to different multicast groups. According | ||
+ | to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses. | ||
+ | Different multicast address regions are designated for different purposes. According to RFC | ||
+ | 2365, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope | ||
+ | address range 239.255.0.0 to 239.255.255.255. | ||
+ | |||
+ | IP multicast packets are delivered by using the Ethernet MAC address range | ||
+ | 01: | ||
+ | 23 bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet | ||
+ | address space. This means that there is ambiguity in delivering packets. If two hosts on the | ||
+ | same subnet each subscribe to a different multicast group whose address differs only in the first | ||
+ | 5 bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the | ||
+ | network software in the hosts to discard the packets which are not required. If multiple multicast | ||
+ | addresses are used, be careful to choose multicast addresses that will result in different Ethernet | ||
+ | multicast addresses. | ||
+ | |||
+ | Multicast data is filtered by the Ethernet controller hardware, only passing subscribed | ||
+ | packets to the software driver for decoding. This improves performance under high network | ||
+ | traffic loads. Ethernet controllers only have a limited number of multicast addresses they can | ||
+ | filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to | ||
+ | more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is | ||
+ | passed to the software driver for filtering, negating the benefit of multicast filtering in hardware. | ||
+ | The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but | ||
+ | some software implementation treat this as a signed value with a maximum value of 2^15 or | ||
+ | 32768. Because of this, the maximum size of a Chapter 10 streaming packet should be no more | ||
+ | than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the | ||
+ | largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it | ||
+ | will be fragmented into smaller packets by the IP software driver before sending them over the | ||
+ | underlying physical network. The fragmented UDP packets are then reassembled into a larger | ||
+ | packet by the IP software driver at the receiving end. There is a performance penalty for this | ||
+ | fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet | ||
+ | small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum | ||
+ | size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer | ||
+ | Ethernet technologies support larger jumbo frames. | ||
+ | |||
+ | Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size will prepend the non-segmented UDP transfer header shown below. | ||
+ | |||
+ | < | ||
+ | struct SuUdpTransferHeaderNonseg | ||
+ | { | ||
+ | uint32_t uVersion | ||
+ | uint32_t uType : | ||
+ | uint32_t uUdpSeqNum | ||
+ | }; | ||
+ | </ | ||
+ | < | ||
+ | |||
+ | A Chapter 10 data packet larger than the 32k maximum size will need to be segmented before transmission, | ||
+ | will prepend the segmented UDP transfer header shown below. IPv6 supports large data packets, negating the need for segmented data packets. | ||
+ | |||
+ | < | ||
+ | struct SuUdpTransferHeaderSeg | ||
+ | { | ||
+ | uint32_t uVersion | ||
+ | uint32_t uType : | ||
+ | uint32_t uUdpSeqNum | ||
+ | uint32_t uChanID | ||
+ | uint32_t uChanSeqNum : 8; // Channel sequence number | ||
+ | uint32_t uReserved | ||
+ | uint32_t uSegOffset; | ||
+ | }; | ||
+ | </ | ||
+ | < | ||
+ | Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a | ||
+ | network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets | ||
+ | will have contiguous sequence numbers for error detection. They should be ignored, though, | ||
+ | when received. |
ch10_handbook/data_file_interpretation.txt · Last modified: 2014/07/16 15:18 by bob