User Tools

Site Tools


ch10_handbook:data_file_interpretation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
ch10_handbook:data_file_interpretation [2014/04/13 12:09] bobch10_handbook:data_file_interpretation [2014/07/16 15:18] (current) – Added links to TSPI and CAN Bus pages bob
Line 1: Line 1:
 +~~ODT~~
 ===== DATA FILE INTERPRETATION ===== ===== DATA FILE INTERPRETATION =====
  
Line 71: Line 72:
 ==== Overall Data Packet Organization ==== ==== Overall Data Packet Organization ====
  
- Overall data packet organization is shown in Figure 6-1. Data packets contain a standard +Overall data packet organization is shown below. Data packets contain a standard 
 header, a data payload containing one or multiple data messages, and a standard trailer. The  header, a data payload containing one or multiple data messages, and a standard trailer. The 
 standard header is composed of a required header, optionally followed by a secondary header.  standard header is composed of a required header, optionally followed by a secondary header. 
Line 100: Line 101:
 | DATA CHECKSUM | **Packet Trailer** | | DATA CHECKSUM | **Packet Trailer** |
  
-**Data Packet Organization**+<html><center><b>Data Packet Organization</b></center></html>
  
 Data packets must contain data. They are not allowed to only contain filler. Filler can be  Data packets must contain data. They are not allowed to only contain filler. Filler can be 
Line 136: Line 137:
  The packet header contains information about the data payload such as time, packet   The packet header contains information about the data payload such as time, packet 
 length, data type, data version, and other information. The layout of a Chapter 10 packet header  length, data type, data version, and other information. The layout of a Chapter 10 packet header 
-is shown in Figure 6-2FIXME+is shown below.
  
 <code> <code>
 struct SuI106Ch10Header  struct SuI106Ch10Header 
     {      { 
-    uint16_t uSync;         // Packet Sync Pattern  +    uint16_t    uSync;              // Packet Sync Pattern  
-    uint16_t uChID;         // Channel ID  +    uint16_t    uChID;              // Channel ID  
-    uint32_t ulPacketLen;   // Total packet length  +    uint32_t    ulPacketLen;        // Total packet length  
-    uint32_t ulDataLen;     // Data length  +    uint32_t    ulDataLen;          // Data length  
-    uint8_t ubyDataVer;     // Data Version  +    uint8_t     ubyDataVer;         // Data Version  
-    uint8_t ubySeqNum;      // Sequence Number  +    uint8_t     ubySeqNum;          // Sequence Number  
-    uint8_t ubyPacketFlags; // Packet Flags  +    uint8_t     ubyPacketFlags;     // Packet Flags  
-    uint8_t ubyDataType;    // Data type  +    uint8_t     ubyDataType;        // Data type  
-    uint8_t aubyRelTime[6]; // Reference time  +    uint8_t     aubyRelTime[6];     // Reference time  
-    uint16_t uChecksum;     // Header Checksum +    uint16_t    uChecksum;          // Header Checksum 
     };      }; 
 </code> </code>
-**Packet header structure.** + 
 +<html><center><b>Packet header structure</b></center></html>
  
 The Channel ID field uniquely identifies the source of the data. The value of the  The Channel ID field uniquely identifies the source of the data. The value of the 
Line 197: Line 199:
  The optional secondary header is used to provide an absolute time (i.e. clock time) stamp   The optional secondary header is used to provide an absolute time (i.e. clock time) stamp 
 for data packets. The secondary header time format can be interpreted several ways. The  for data packets. The secondary header time format can be interpreted several ways. The 
-specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure in  +specific interpretation is determined by the value of header Flag Bits 2 and 3. The structure below 
-Figure 6-3 is used when secondary header time is to be interpreted as a Chapter 4 format value  +is used when secondary header time is to be interpreted as a Chapter 4 format value  
-(Flag Bits 3-2 = 0). The structure in Figure 6-4 is used when secondary header time is to be +(Flag Bits 3-2 = 0). The following structure is used when secondary header time is to be 
 interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1). interpreted as an IEEE 1588 format value (Flag Bits 3-2 = 1).
  
Line 213: Line 215:
     };      }; 
 </code>  </code> 
-**Optional secondary header structure with IRIG 106 Ch 4 time representation.**+<html><center><b>Optional secondary header structure with IRIG 106 Ch 4 time representation</b></center></html>
  
- <code>+<code>
 struct SuI106Ch10SecHeader_1588Time  struct SuI106Ch10SecHeader_1588Time 
     {      { 
Line 224: Line 226:
     };      }; 
 </code> </code>
-**Optional secondary header structure with IEEE 1588 time representation.** +<html><center><b>Optional secondary header structure with IEEE 1588 time representation</b></center></html>
  
 ==== Data Payload ==== ==== Data Payload ====
Line 240: Line 242:
 when there are no more data messages to read.  when there are no more data messages to read. 
    
- Intra-packet headers, when they are present, typically contain one, or sometimes more +Intra-packet headers, when they are present, typically contain one, or sometimes more 
 than one, time stamp as well as other information about the data message that follows.  than one, time stamp as well as other information about the data message that follows. 
-Commonly used structures for intra-packet time data are shown in Figure 6-5, Figure 6-6, and  +Commonly used structures for intra-packet time data are shown in the three figures below. 
-Figure 6-7. These three time structures will be referenced in most of the data format descriptions +These three time structures will be referenced in most of the data format descriptions 
 that follow.  that follow. 
  
Line 253: Line 255:
     };      }; 
 </code> </code>
-**Intra-packet Time Stamp, 48-bit RTC.** +<html><center><b>Intra-packet Time Stamp, 48-bit RTC</b></center></html>
    
 <code> <code>
Line 264: Line 266:
     };      }; 
 </code> </code>
-**Intra-packet Time Stamp, IRIG 106 Ch 4 binary**+<html><center><b>Intra-packet Time Stamp, IRIG 106 Ch 4 binary</b></center></html>
  
 <code> <code>
Line 273: Line 275:
     };      }; 
 </code> </code>
-**Intra-packet Time Stamp, IEEE 1588**+<html><center><b>Intra-packet Time Stamp, IEEE 1588</b></center></html>
  
-Computer Generated Data+==== Data Formats ==== 
 + 
 +[[Computer Generated Data]] 
 + 
 +[[PCM Data]] 
 + 
 +[[Time Data]] 
 + 
 +[[MIL-STD-1553 Data]] 
 + 
 +[[Analog Data]] 
 + 
 +[[Discrete Data]] 
 + 
 +[[Message Data]] 
 + 
 +[[ARINC 429 Data]] 
 + 
 +[[Video Data]] 
 + 
 +[[Image Data]] 
 + 
 +[[UART Data]] 
 + 
 +[[IEEE 1394 Data]] 
 + 
 +[[Parallel Data]] 
 + 
 +[[Ethernet Data]] 
 + 
 +[[TSPI Data]] 
 + 
 +[[CAN Bus Data]] 
 + 
 +==== Time Interpretation ==== 
 + 
 + Chapter 10 defines a 48-bit Relative Time Counter (RTC) as the basis for all packet and  
 +message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100  
 +nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it  
 +could be initialized to zero or some random number. Some recorder vendors will preset the RTC  
 +to a value based on absolute clock time, but for interoperability reasons it is unwise to assume  
 +the RTC value will be related to absolute clock time in any meaningful fashion.  
 +Absolute clock time comes into a recorder and is recorded much like any other data source. In  
 +fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to  
 +record input time signals. Since Time Data, Format 1 packets contain both the absolute input  
 +time value and the RTC clock value at the instant the absolute time was valid, these packets can  
 +be used to relate RTC values to the input absolute time source. For example, if a time packet is  
 +recorded with a RTC value of 1,000,000 and an absolute time value of 100:12:30:25.000, then  
 +the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to  
 +be 100:12:30:25.015 (150,000 clock tics x 100 nsec per tic = 15 msec).  
 +  
 + When multiple time channels are available, it is incumbent on the programmer or data  
 +analyst to determine and select the best source of time for a particular data set. For example,  
 +there may be separate time channels for time derived from IRIG B, Global Positioning System  
 +(GPS), and an internal battery backed up clock. In this scenario, all of these time sources are  
 +present in the data file as separate channels, each correlating the RTC to its own notion of clock  
 +time. The software application may allow the user to select which source of time to use for a  
 +given analysis. Alternatively, the software may decide the “best” source of time, depending on  
 +which time channels are providing valid time. In general, each time source will provide a  
 +slightly (or not so slightly) different clock time. It is usually most correct to select one time  
 +channel only and to use this channel exclusively to correlate RTC time to absolute clock time for  
 +all data packet types.  
 +  
 + The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least  
 +as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on  
 +the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It 
 +is tempting for an application to find a single time packet early in a recording and to use those  
 +time values to subsequently derive clock time from relative time. It is better to use the clock and  
 +relative time values from a time packet that occurs near the current data packet as the data file is  
 +decoded since there is some drift in the RTC during a recording session. It also may be the case  
 +that there is a jump in input clock time during a recording, such as when GPS locks for the first  
 +time, or when an IRIG time source is reprogrammed. 
 + 
 +Index and Event Records  
 + Often times it is useful to make an in-memory version of the data file index. This allows  
 +rapid access to recorded data packets based on time or the occurrence of events. A general  
 +algorithm for reading all root and node index packets is as follows:  
 +  
 +1. If "R-x\IDX\E" is not equal to "T" then index does not exist. 
 + 
 +2. Move read pointer to last packet of data file. Store file offset of this packet.  
 + 
 +3. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3)  
 +then index does not exist. 
 + 
 +4. Get the index count from the CSDW.  
 + 
 +5. For each root index contained in the packet,  
 +  * Read the Node Index offset value  
 +  * Move the read pointer to the Node Index offset value  
 +  * Read the Node Index packet  
 +  * Get the node index count from the CSDW  
 +  * For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values.  
 + 
 +6. Read last root node index. If offset value is equal to current root node packet offset  
 +(stored in Step 2) then done.  
 + 
 +7. Else the move read pointer to the next Root Index packet offset value  
 + 
 +8. Read the next Root Index packet.  
 + 
 +9. Go to Step 4.  
 + 
 +==== Data Streaming ==== 
 +  
 + Chapter 10 recorders can stream their data over one of their download interface network  
 +ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is  
 +normally done over an Ethernet port, but any network connection that supports UDP/IP can use  
 +this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the  
 +use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more  
 +specific unicast and multicast IP addresses, or broadcast address. Different channels can be  
 +addressed to different addresses.  
 +  
 + It is common to publish different groups of data to different multicast groups. According  
 +to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses.  
 +Different multicast address regions are designated for different purposes. According to RFC  
 +2365, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope  
 +address range 239.255.0.0 to 239.255.255.255.  
 + 
 +IP multicast packets are delivered by using the Ethernet MAC address range  
 +01:00:5e:00:00:00 - 01:00:5e:7f:ff:ff. This is 23 bits of available address space. The lower  
 +23 bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet  
 +address space. This means that there is ambiguity in delivering packets. If two hosts on the  
 +same subnet each subscribe to a different multicast group whose address differs only in the first  
 +5 bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the  
 +network software in the hosts to discard the packets which are not required. If multiple multicast  
 +addresses are used, be careful to choose multicast addresses that will result in different Ethernet  
 +multicast addresses.  
 +  
 +Multicast data is filtered by the Ethernet controller hardware, only passing subscribed  
 +packets to the software driver for decoding. This improves performance under high network  
 +traffic loads. Ethernet controllers only have a limited number of multicast addresses they can  
 +filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to  
 +more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is  
 +passed to the software driver for filtering, negating the benefit of multicast filtering in hardware.  
 +The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but  
 +some software implementation treat this as a signed value with a maximum value of 2^15 or  
 +32768. Because of this, the maximum size of a Chapter 10 streaming packet should be no more  
 +than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the  
 +largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it  
 +will be fragmented into smaller packets by the IP software driver before sending them over the  
 +underlying physical network. The fragmented UDP packets are then reassembled into a larger  
 +packet by the IP software driver at the receiving end. There is a performance penalty for this  
 +fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet  
 +small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum  
 +size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer  
 +Ethernet technologies support larger jumbo frames.  
 +  
 +Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size will prepend the non-segmented UDP transfer header shown below.  
 + 
 +<code> 
 +struct SuUdpTransferHeaderNonseg  
 +    {  
 +    uint32_t uVersion    :  4;  // Version  
 +    uint32_t uType       :  4;  // Type of message  
 +    uint32_t uUdpSeqNum  : 24;  // UDP sequence number  
 +    };  
 +</code> 
 +<html><center><b>UDP Transfer Header, non-segmented data</b></center></html> 
 + 
 +A Chapter 10 data packet larger than the 32k maximum size will need to be segmented before transmission, and  
 +will prepend the segmented UDP transfer header shown below. IPv6 supports large data packets, negating the need for segmented data packets.  
 + 
 +<code> 
 +struct SuUdpTransferHeaderSeg  
 +    {  
 +    uint32_t uVersion    :  4;  // Version  
 +    uint32_t uType       :  4;  // Type of message  
 +    uint32_t uUdpSeqNum  : 24;  // UDP sequence number  
 +    uint32_t uChanID     : 16;  // Channel ID  
 +    uint32_t uChanSeqNum :  8;  // Channel sequence number  
 +    uint32_t uReserved   :  8;  //  
 +    uint32_t uSegOffset;        // Segment offset  
 +    };  
 +</code> 
 +<html><center><b>UDP Transfer Header, segmented data</b></center></html>
  
 +Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a 
 +network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets 
 +will have contiguous sequence numbers for error detection. They should be ignored, though, 
 +when received. 
ch10_handbook/data_file_interpretation.1397408968.txt.gz · Last modified: 2014/04/13 12:09 by bob

Except where otherwise noted, content on this wiki is licensed under the following license: CC0 1.0 Universal
CC0 1.0 Universal Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki