Differences

This shows you the differences between two versions of the page.

--- ch10_handbook:data_file_interpretation [2014/04/13 12:09] – bob
+++ ch10_handbook:data_file_interpretation [2014/04/13 17:10] – bob
@@ Line 275: / Line 275: @@
 **Intra-packet Time Stamp, IEEE 1588**
-Computer Generated Data
+==== Data Formats ====
+[[Computer Generated Data]]
+[[PCM Data]]
+[[Time Data]]
+[[MIL-STD-1553 Data]]
+[[Analog Data]]
+[[Discrete Data]]
+[[Message Data]]
+[[ARINC 429 Data]]
+[[Video Data]]
+[[Image Data]]
+[[UART Data]]
+[[IEEE 1394 Data]]
+[[Parallel Data]]
+[[Ethernet Data]]
+==== Time Interpretation ====
+ Chapter 10 defines a 48-bit Relative Time Counter (RTC) as the basis for all packet and
+message time stamps. The RTC clock is 10 MHz, resulting in a clock resolution of 100
+nanoseconds. There is no constraint on the absolute value of the RTC; at recorder power on, it
+could be initialized to zero or some random number. Some recorder vendors will preset the RTC
+to a value based on absolute clock time, but for interoperability reasons it is unwise to assume
+the RTC value will be related to absolute clock time in any meaningful fashion.
+Absolute clock time comes into a recorder and is recorded much like any other data source. In
+fact, there may be multiple time sources recorded. Time Data, Format 1 data packets are used to
+record input time signals. Since Time Data, Format 1 packets contain both the absolute input
+time value and the RTC clock value at the instant the absolute time was valid, these packets can
+be used to relate RTC values to the input absolute time source. For example, if a time packet is
+recorded with a RTC value of 1,000,000 and an absolute time value of 100:12:30:25.000, then
+the clock time of a subsequent data packet with an RTC value of 1,150,000 could be deduced to
+be 100:12:30:25.015 (150,000 clock tics x 100 nsec per tic = 15 msec).
+ When multiple time channels are available, it is incumbent on the programmer or data
+analyst to determine and select the best source of time for a particular data set. For example,
+there may be separate time channels for time derived from IRIG B, Global Positioning System
+(GPS), and an internal battery backed up clock. In this scenario, all of these time sources are
+present in the data file as separate channels, each correlating the RTC to its own notion of clock
+time. The software application may allow the user to select which source of time to use for a
+given analysis. Alternatively, the software may decide the “best” source of time, depending on
+which time channels are providing valid time. In general, each time source will provide a
+slightly (or not so slightly) different clock time. It is usually most correct to select one time
+channel only and to use this channel exclusively to correlate RTC time to absolute clock time for
+all data packet types.
+ The stability of the RTC isn’t specified in Chapter 10 other than to require it to be at least
+as good as a common crystal oscillator. A good grade crystal oscillator can provide stability on
+the order of 10 ppm. Some vendors provide a RTC source considerably more stable than this. It
+is tempting for an application to find a single time packet early in a recording and to use those
+time values to subsequently derive clock time from relative time. It is better to use the clock and
+relative time values from a time packet that occurs near the current data packet as the data file is
+decoded since there is some drift in the RTC during a recording session. It also may be the case
+that there is a jump in input clock time during a recording, such as when GPS locks for the first
+time, or when an IRIG time source is reprogrammed.
+Index and Event Records
+ Often times it is useful to make an in-memory version of the data file index. This allows
+rapid access to recorded data packets based on time or the occurrence of events. A general
+algorithm for reading all root and node index packets is as follows:
+. If "R-x\IDX\E" is not equal to "T" then index does not exist.
+. Move read pointer to last packet of data file. Store file offset of this packet.
+. If last packet data type does not equal 0x03 (Computer Generated Data, Format 3)
+then index does not exist.
+. Get the index count from the CSDW.
+. For each root index contained in the packet,
+  * Read the Node Index offset value
+  * Move the read pointer to the Node Index offset value
+  * Read the Node Index packet
+  * Get the node index count from the CSDW
+  * For each node index contained in the packet read and store the time stamp, channel ID, data type, and data packet offset values.
+. Read last root node index. If offset value is equal to current root node packet offset
+(stored in Step 2) then done.
+. Else the move read pointer to the next Root Index packet offset value
+. Read the next Root Index packet.
+. Go to Step 4.
+==== Data Streaming ====
+ Chapter 10 recorders can stream their data over one of their download interface network
+ports using User Datagram Protocol (UDP)/IP and Chapter 10 UDP transfer headers. This is
+normally done over an Ethernet port, but any network connection that supports UDP/IP can use
+this method. The .PUBLISH command is used to control data streaming. Chapter 6 defines the
+use of .PUBLISH and has numerous examples of its use. Data can be streamed to one or more
+specific unicast and multicast IP addresses, or broadcast address. Different channels can be
+addressed to different addresses.
+ It is common to publish different groups of data to different multicast groups. According
+to RFC 3171, addresses 224.0.0.0 to 239.255.255.255 are designated as multicast addresses.
+Different multicast address regions are designated for different purposes. According to RFC
+, Chapter 10 data streaming should be directed to multicast addresses in the Local Scope
+address range 239.255.0.0 to 239.255.255.255.
+IP multicast packets are delivered by using the Ethernet MAC address range
+:00:5e:00:00:00 - 01:00:5e:7f:ff:ff. This is 23 bits of available address space. The lower
+bits of the 28-bit multicast IP address are mapped into the 23 bits of available Ethernet
+address space. This means that there is ambiguity in delivering packets. If two hosts on the
+same subnet each subscribe to a different multicast group whose address differs only in the first
+bits, Ethernet packets for both multicast groups will be delivered to both hosts, requiring the
+network software in the hosts to discard the packets which are not required. If multiple multicast
+addresses are used, be careful to choose multicast addresses that will result in different Ethernet
+multicast addresses.
+ Multicast data is filtered by the Ethernet controller hardware, only passing subscribed
+packets to the software driver for decoding. This improves performance under high network
+traffic loads. Ethernet controllers only have a limited number of multicast addresses they can
+filter. 16 multicast addresses is a common hardware limit. If a workstation needs to subscribe to
+more multicast addresses than the Ethernet hardware provides for, then all multicast traffic is
+passed to the software driver for filtering, negating the benefit of multicast filtering in hardware.
+The size of a UDP packet is represented by a 16 bit value in the IPv4 IP and UDP headers, but
+some software implementation treat this as a signed value with a maximum value of 2^15 or
+. Because of this, the maximum size of a Chapter 10 streaming packet should be no more
+than 32724 bytes. Physical networks have a Maximum Transfer Unit (MTU), which is the
+largest data packet they can carry. If a UDP packet has a size larger than the network MTU, it
+will be fragmented into smaller packets by the IP software driver before sending them over the
+underlying physical network. The fragmented UDP packets are then reassembled into a larger
+packet by the IP software driver at the receiving end. There is a performance penalty for this
+fragmentation and reassembly. Better performance may be achieved by choosing a UDP packet
+small enough to avoid fragmentation and reassembly. Regular Ethernet supports a maximum
+size of 1500 bytes of data payload (IP header, UDP header, and UDP data) but some newer
+Ethernet technologies support larger jumbo frames.
+ Chapter 10 data packets are sent in a UDP/IP packet by prepending a UDP transfer
+header to the UDP data payload. Chapter 10 data packet(s) smaller than the 32k maximum size
+will prepend the non-segmented UDP transfer header shown in Figure 6-54. A Chapter 10 data
+packet larger than the 32k maximum size will need to be segmented before transmission, and
+will prepend the segmented UDP transfer header shown in Figure 6-55. IPv6 supports large data
+packets, negating the need for segmented data packets.
+<code>
+struct SuUdpTransferHeaderNonseg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    };
+</code>
+**UDP Transfer Header, non-segmented data.**
+<code>
+struct SuUdpTransferHeaderSeg
+    {
+    uint32_t uVersion    :  4;  // Version
+    uint32_t uType       :  4;  // Type of message
+    uint32_t uUdpSeqNum  : 24;  // UDP sequence number
+    uint32_t uChanID     : 16;  // Channel ID
+    uint32_t uChanSeqNum :  8;  // Channel sequence number
+    uint32_t uReserved   :  8;  //
+    uint32_t uSegOffset;        // Segment offset
+    };
+</code>
+**UDP Transfer Header, segmented data.**
+Computer Generated Data, Format 3 (Recording Index) packets are meaningless in a
+network data stream. It is necessary that they be transmitted so that Channel ID 0 data packets
+will have contiguous sequence numbers for error detection. They should be ignored, though,
+when received.