warts man page

warts — format for scamper's warts storage.

Description

This document describes the warts binary file format used by scamper(1). A warts file consists of a set of records, each of which begin with an 8 byte object header.

Warts Object Header

The header consists of a 2-byte magic number (0x1205), a 2-byte type value, and a 4-byte length value defining the size of the object that follows. All 16-bit and 32-bit numbers are written in network byte order. The currently defined types, and the types of scamper(1) object they map to, are as follows:

A new type number can be requested by emailing the author of scamper. The structure of each warts record beyond the header is arbitrary, though some conventions have been established to promote extensibility.

Flags and Parameters

The warts routines in scamper provide the ability to conditionally store arbitrary data in a forwards compatible method. A set of flags and parameters begins with a sequence of bytes that denote which items are included. If any flags are set, then after the flags is a 2-byte field that records the length of the parameters that follow. Finally, the data follows. The following figure illustrates how flags are recorded:

   Byte zero           Byte one          Byte two 
 8 7 6 5 4 3 2 1    8 7 6 5 4 3 2 1   8 7 6 5 4 3 2 1 
+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+ 
|1              |  |1              | |0              | 
+-+-+-+-+-+-+-+-+  +-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+

The most significant bit of each byte is the `link' bit; it determines if the next byte in the sequence contains flags. The low-order 7 bits of each byte signal if the corresponding field is written out in the parameters that follow. In the figure, the link bit is set to one in the first two bytes, and zero in the final byte, signifying that three flag-bytes are included.

The rest of each byte is used to record flags, whose position in the sequence signifies if a particular parameter is included. For example, if bit 6 of byte zero is set, then parameter 6 is included, and if bit 5 of byte one is set, then parameter 12 is included, and if bit 2 of byte two is set, then parameter 16 is included.

Addresses

A warts file may have addresses embedded in two ways. The first is deprecated: the address is written as a data object that can be globally referenced before the data object that uses it is written. A reader therefore must keep a copy of all addresses it reads in order to be able to decode data objects that subsequently reference it, which can consume a significant amount of memory. The format of the address is

  Warts header   ID Modulo    Type     Address 
+--------------+-----------+--------+----//----+ 
| 8 bytes      |  1 byte   | 1 byte |          | 
| type = 5     |           |        |          | 
+--------------+-----------+--------+----//----+

A reader determines the ID number of each address by the order in which it appears, and can sanity check the ID number it determines by comparing the lower 8 bits of the computed ID with the ID that is embedded in the record. Address ID numbers start at one; zero is reserved for when no address is embedded. The type corresponds to the type of address that follows. The currently defined types are as follows:

The second method to embed an address is to embed the address in each data object that requires that address. The format of that address can take one of two forms, depending on whether or not the address is being defined or referenced. A defined address declares a new address that has scope for the data object being embedded; a reader adds the address to the end of a table so that it can be later referenced without having to re-define the address. In this method, ID numbers start from zero. The format of a defined address is:

  Address length    Type      Address 
+----------------+---------+----//-----+ 
|  uint8_t       | uint8_t |           | 
|  value > 0     |         |           | 
+----------------+---------+-----------+

The format of a referenced address is:

  Magic value     ID number 
+--------------+------------+ 
|  uint8_t     |  uint32_t  | 
|  value == 0  |            | 
+--------------+------------+

Embedding Other Types

Bytes, unsigned 16-bit integers, and unsigned 32 bit integers are embedded directly, using network byte order where appropriate. ASCII strings are also embedded directly, including the trailing null byte to terminate the string. Time values (timeval) are embedded as two unsigned 32 bit integers; the first number counts the number of seconds that have elapsed since the Unix epoch, the second number counts the number of microseconds that have elapsed in the current second. Round-trip-time (RTT) values are embedded as a single unsigned 32 bit integer that counts the number of microseconds that elapsed.

Several measurements record ICMP extension data, so there is a standardised method to record a set of ICMP extensions. Individual ICMP extension records are written in the following format: The format of a list structure is:

A set of ICMP extension records is written in the following format:

  Total Length   Extension #1 .. Extension #N 
+--------------+------//------+-------//------+ 
|  uint16_t    |              |               | 
|              |              |               | 
+--------------+------//------+-------//------+

List Structure

The format of a list structure is:

The List ID assigned by warts is subsequently used by objects that reference the list to identify which list they refer to.

Cycle Structure

Three types of cycle records may be written: a start record denoting the starting point for a new cycle, a definition record declaring a cycle record whose corresponding start record is in a different file, and a cycle stop record, denoting the end point for a cycle. The format of the cycle start and definition structures is:

The format of the cycle stop structure is:

Traceroute Structure

Traceroute structures consist of traceroute parameters, hop records, and an optional series of additional data types for special types of traceroute invokation. The general form of a traceroute recorded in warts is as follows:

The flags and data types that describe traceroute are as follows:

The traceroute flags field has the following fields:

Hop records are written in series. Each hop record takes the following form:

Optional traceroute data, such as PMTUD and doubletree control and result structures are included after hop records. Optional traceroute data begins with a 16-bit header; the first four bits define the type of record, and the remaining 12 bits specify the length of the record. Currently, three types of optional data are defined: PMTUD data (1), Last-ditch probing results (2), and doubletree (3).

The format of the last-ditch data is:

The format of PMTUD data is:

The format of the PMTUD flags and attributes is:

The format of the PMTUD notes is:

The tree types of PMTUD notes are: ordinary PTB (1), PTB with invalid next-hop MTU (2), and an inferred MTU in the absence of a PTB (3).

See Also

scamper(1), libscamperfile(3), sc_wartsdump(1),

M. Luckie, Scamper: a Scalable and Extensible Packet Prober for Active Measurement of the Internet, Proc. ACM/SIGCOMM Internet Measurement Conference 2010.

Authors

warts is written by Matthew Luckie <mjl@luckie.org.nz>.

Referenced By

sc_filterpolicy(1), sc_tbitblind(1).

May 16, 2011