performance metrics archive format


Performance Co-Pilot (PCP) archives store historical values about arbitrary metrics recorded from a single host. Archives are machine independent and self-contained - all metadata required for off-line or off-site analysis is stored.

The format is stable in order to allow long-term historical storage and processing by PMAPI client tools.

Archives may be read by most PCP client tools, using the -a/--archive NAME option, or dumped raw by pmdumplog(1). Archives are created primarily by pmlogger(1), however they can also be created using the LOGIMPORT(3) programming interface.

Archives may be merged, analyzed, modified and subsampled using pmlogreduce(1), pmlogsummary(1), pmlogrewrite(1) and pmlogextract(1). In addition, PCP archives may be examined in sets or grouped together into "archive folios", which are created and managed by the mkaf(1) and pmafm(1) tools.

Archives consist of several physical files that share a common arbitrary prefix, e.g. myarchive.

myarchive.0, myarchive.1, ...

One or more data volumes containing the metric values and any error codes encountered during metric sampling. Typically the largest of the files and may grow very rapidly, depending on the pmlogger sampling interval(s) being used.


Information for PMAPI functions such as pmLookupDesc(3), pmLookupLabels(3) and pmLookupInDom(3). The metadata file may grow sporadically as logged metrics, instance domains and labels vary over time.


A temporal index, mapping timestamps to offsets in the other files.

Common Features

All three types of files have a similar record-based structure, a convention of network-byte-order (big-endian) encoding, and 32-bit fields for tagging/padding for those records. Strings are stored as 8-bit characters without assuming a specific encoding, so normally ASCII. See also the __pmLog* types in libpcp.h.

Record Framing

The volume and meta files are divided into self-identifying records.

04N, length of record, in bytes, including this field
4N-8record payload, usually starting with a 32-bit tag
N-44N, length of record (again)

Archive Log Label

All three types of files begin with a "log label" header, which identifies the host name, the time interval covered, and a time zone.

04tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602
44pid of pmlogger process that wrote file
84log start time, seconds part (past UNIX epoch)
124log start time, microseconds part
164current log volume number (or -1=.meta, -2=.index)
2064name of collection host
8040time zone string ($TZ environment variable)

All fields, except for the current log volume number field, match for all archive-related files produced by a single run of the tool.

Archive Volume (.0, .1, ...) Records


After the archive log label record, an archive volume file contains metric values corresponding to the pmResult set of one pmFetch operation, which is almost identical to the form on disk. The record size may vary according to number of PMIDs being fetched, the number of instances for their domains. File size is limited to 2GiB, due to storage of 32-bit offsets within the temporal index.

04timestamp, seconds part (past UNIX epoch)
44timestamp, microseconds part
84number of PMIDs with data following
12MpmValueSet #0
12+MNpmValueSet #1
NOPXpmValueBlock #0
NOP+XYpmValueBlock #1

Records with a number-of-PMIDs equal to zero are "mark records", and represent interruptions, missing data, or time discontinuities in logging.


This subrecord represents the measurements for one metric.

44number of values
84storage mode, PM_VAL_INSITU=0 or PM_VAL_DPTR=1
12MpmValue #0
12+MNpmValue #1

The metric-description metadata for PMIDs is found in the .meta files. These entries are not timestamped, so the metadata is assumed to be unchanging throughout the archiving session.


This subrecord represents one measurement for one instance of the metric. It is a variant type, depending on the parent pmValueSet's value-format field.  This allows small numbers to be encoded compactly, but retain flexibility for larger or variable-length data to be stored later in the pmResult record.

04number in instance-domain (or PM_IN_NULL=-1)
44value (INSITU) or
offset in pmResult to our pmValueBlock (DPTR)

The instance-domain metadata for PMIDs is found in the .meta files. Since the numeric mappings may change during the lifetime of the logging session, it is important to match up the timestamp of the measurement record with the corresponding instance-domain record. That is, the instance-domain corresponding to a measurement at time T are the records with largest timestamps T' <= T.


Instances of this subrecord are placed at the end of the pmValueSet, after all the pmValue subrecords. If (and only if) needed, they are padded at the end to the next-higher 32-bit boundary.

01value type (same as pmDesc.type)
134 + N, the length of the subrecord
4Nbytes that make up the raw value
4+N0-3padding (not included in the 4+N length field)

Note that for PM_TYPE_STRING, the length includes an explicit NULL terminator byte. For PM_TYPE_EVENT, the value bytestring is further structured.


After the archive log label record, the metadata file contains interleaved metric-description and timestamped instance-domain descriptors. File size is limited to 2GiB, due to storage of 32-bit offsets within the temporal index. Unlike the data volumes, these records are not forced to 32-bit alignment. See also libpcp/logmeta.c.


Instances of this record represent the metric description, giving a name, type, instance-domain identifier, and a set of names to each PMID used in the archive volume.

04tag, TYPE_DESC=1
84type (PM_TYPE_*)
124instance domain number
164semantics of value (PM_SEM_*)
204units: bit-packed pmUnits
44number of alternative names for this PMID
284N: number of bytes in this name
32Nbytes of the name, no NULL terminator nor padding
32+N4N2: number of bytes in next name
36+NN2bytes of the name, no NULL terminator nor padding


Instances of this record represent the number-string mapping table of an instance domain. The instance domain number will have already been mentioned in a prior pmDesc record. As new instances may appear over a long archiving run these records are timestamped, and must be searched when decoding pmResult records from the archive data volumes. Instance names may be reused between instance numbers, so an offset-based string table is used that facilitates sharing.

04tag, TYPE_INDOM=2
44timestamp, seconds part (past UNIX epoch)
84timestamp, microseconds part
124instance domain number
164N: number of instances in domain, normally >0
204first instance number
244second instance number (if appropriate)
20+4*N4first offset into string table (see below)
20+4*N+44second offset into string table (etc.)
20+8*NMbase of string table, containing
packed, NULL-terminated instance names

Records of this form replace the existing instance-domain: prior records are not searched for resolving instance numbers in measurements after this timestamp.


Instances of this record represent sets of name:value pairs associated with labels of the context, instance domains and individual performance metrics - refer to pmLookupLabels(3) for further details.

Any instance domain number will have already been mentioned in a prior pmDesc record. As new labels can appear during an archiving session, these records are timestamped and must be searched when decoding pmResult records from the archive data volumes.

04tag, TYPE_LABEL=3
44timestamp, seconds part (past UNIX epoch)
84timestamp, microseconds part
124label type (PM_LABEL_* type macros.)
164numeric identifier - domain, PMID, etc
or PM_IN_NULL=-1 for context labels
204N: number of label sets in this record,
usually 1 except in the case of instances
244offset to the start of the JSONB labels string
28L1first labelset array entry (see below)
28+L1LNN-th labelset array entry (see below)
28+L1+...LNMconcatenated JSONB strings for all labelsets

Records of this form replace the existing labels for a given type: prior records are not searched for resolving that class of label in measurements after this timestamp.

The individual labelset array entries are variable length, depending on the number of labels present within that set. These entries contain the instance identifiers (in the case of type PM_LABEL_INSTANCES labels), lengths and offsets of each label name and value, and also any flags set for each label.

04instance identifier (or PM_IN_NULL=-1)
44length of JSONB label string
84N: number of labels in this labelset
122first label name offset
141first label name length
151first label flags (e.g. optionality)
162first label value offset
182first label value length
202second label name offset (if appropriate)


This record stores help text associated with a metric or an instance domain - as provided by pmLookupText(3) and pmLookupInDomText(3).

The metric identifier and instance domain number will have already been mentioned in a prior pmDesc record.

04tag, TYPE_TEXT=4
44text and identifier type (PM_TEXT_* macros.)
84numeric identifier - PMID or instance domain
12Mhelp text string, arbitrary text


After the archive log label record, the temporal index file contains a plainly concatenated, unframed group of tuples, which relate timestamps to 32-bit seek offsets in the volume and meta files. These records are fixed-size, fixed-format, and are not enclosed in the standard length/payload/length wrapper: they take up the entire remainder of the .index file. See also libpcp/logutil.c.

04event time, seconds part (past UNIX epoch)
44event time, microseconds part
84archive volume number (0...N)
124byte offset in .meta file of pmDesc or pmLogIndom
164byte offset in archive volume file of pmResult

Since the temporal index is optional, and exists only to speed up time-based random access to metrics and their metadata, the index records are emitted only intermittently. An archive reader program should not presume any particular rate of data flow into the index. However, common events that may trigger a new temporal-index record include changes in instance-domains, switching over to a new archive volume, and starting or stopping logging. One reliable invariant however is that, for each index entry, there are to be no meta or archive-volume records with a timestamp after that in the index, but physically before the byte-offset in the index.


Several PCP tools create archives in standard locations:


default location for the interactive chart recording mode in pmchart(1)


default location for pmlogger_daily(1) and pmlogger_check(1) scripts

See Also

PCPIntro(1), PMAPI(3), pmLookupDesc(3), pmLookupInDom(3), pmLookupInDomText(3), pmLookupLabels(3), pmLookupText(3), mkaf(1), pmafm(1), pmchart(1), pmdumplog(1), pmlogger(1), pmlogger_check(1), pmlogger_daily(1), pmlogreduce(1), pmlogrewrite(1), pmlogsummary(1), pcp.conf(5), and pcp.env(5).

Referenced By

pcp2elasticsearch(1), pcp2graphite(1), pcp2influxdb(1), pcp2json(1), pcp2spark(1), pcp2xlsx(1), pcp2xml(1), pcp2zabbix(1), PCPIntro(1), pmrep(1).

Performance Co-Pilot