rteval - Man Page

evaluate latency performance of a realtime Linux system

The rteval program is used to evaluate the realtime performance of a Linux kernel on a particular hardware system. It runs measurement modules (cyclictest or timerlat) to measure scheduling and timer latency while simultaneously running system load modules (kernel compile, hackbench, stress-ng) to stress the system.

The program runs for a specified duration (default 60 seconds, typically 12+ hours for production testing) and generates an XML report containing system information, raw measurement data, and statistical analysis including min/max latency, standard deviation, and histograms.

Results are stored in a timestamped directory (rteval-YYYYMMDD-N) containing the XML report and optional logs.

Options

General Options

-h, --help: Display help message and exit.
-V, --version: Print rteval version and exit.
-d, --duration DURATION: Specify length of test run in seconds (default: 60.0). Can be specified as a floating point number (e.g., 1.5 for 90 seconds) or with time suffixes.
-f, --inifile FILE: Initialization file for configuring loads and behavior.
-w, --workdir DIRECTORY: Top directory for rteval data (default: current directory).
-i, --installdir DIRECTORY: Place to locate installed templates (default: /usr/share/rteval).
-l, --loaddir DIRECTORY: Directory for load source tarballs (default: ./loadsource).
-a, --annotate STRING: Add an annotation string which is stored in the report.
-D, --debug: Turn on debug prints (default: False).
-L, --logging: Log the output of measurement and load modules in the report directory.
-q, --quiet: Turn on quiet mode (default: False).
-v, --verbose: Turn on verbose prints (default: False).
-s, --sysreport: Run sysreport to collect system data (default: False).

Load Control Options

--noload: Only run the measurements (don't run load modules).
-O, --onlyload: Only run the loads (don't run measurement threads).
--loads-cpulist CPULIST: CPU list where load modules will run (e.g., 0-3,8-11).

CPU Isolation and Cpuset Options

--cpusets: Use cgroup v2 cpusets to isolate measurement and housekeeping workloads.
--housekeeping CPULIST: Isolated CPUs reserved for system tasks (not used by rteval).
--warn-non-isolated-core-sharing: Warn about measurement and load CPUs sharing cores even when neither is isolated.
--cleanup-cpusets: Remove any leftover rteval cpusets from previous runs and exit.

Measurement Module Options

--measurement-module MODULE: Select measurement module: cyclictest or timerlat (overrides config file).
--measurement-cpulist CPULIST: CPU list where measurement modules will run.
--measurement-run-on-isolcpus: Include isolated CPUs in default cpulist.
--measurement-idle-set IDLESTATE: Idle state depth to set on CPUs running measurement modules.

Cyclictest Module Options

--cyclictest-interval INTV_US: Base interval of the threads in microseconds (default: 100).
--cyclictest-buckets NUM: Histogram width (default: 3500).
--cyclictest-priority PRIO: Run cyclictest with the given priority (default: 95).
--cyclictest-breaktrace USEC: Send a break trace command when latency > USEC.
--cyclictest-threshold USEC: Exit rteval if latency > USEC.

Timerlat Module Options

--timerlat-interval INTV_US: Base interval or period of threads in microseconds (default: 100).
--timerlat-priority PRIO: Run rtla timerlat with this priority (default: 95).
--timerlat-buckets NUM: Number of histogram buckets (default: 3500).
--timerlat-stoptrace USEC: Stop trace if thread latency higher than USEC.
--timerlat-trace FILE: File to save trace to.
--timerlat-dma_latency USEC: Set /dev/cpu_dma_latency to USEC (set to None to disable) (default: 0).

Hackbench Load Module Options

--hackbench-jobspercore NUM: Number of working threads per CPU core (default: 5).
--hackbench-runlowmem True|False: Run hackbench on machines where low memory is detected.

Kcompile Load Module Options

--kcompile-source TARBALL: Source tar ball (default: linux-6.17.7.tar.xz).
--kcompile-jobspercore NUM: Number of working threads per core (default: 2).

Stress-ng Load Module Options

--stressng-option OPTION: Stressor specific option.
--stressng-arg ARG: Stressor specific arg.
--stressng-timeout T: Timeout after T seconds.

Report Analysis Options

-H, --raw-histogram XMLFILE [XMLFILE ...]: Generate raw histogram data for one or more already existing XML reports.
-Z, --summarize XMLFILE [XMLFILE ...]: Summarize one or more already existing XML reports.
-S, --source-download [KERNEL_VERSION]: Download a source kernel from kernel.org and exit.

Examples

Basic Usage

Run a 60-second test with default settings (cyclictest measurement, kernel compile and hackbench loads):

    sudo rteval

Long Duration Test

Run a 12-hour test with timerlat measurement:

    sudo rteval -d 43200 --measurement-module timerlat

Measurement Only

Run timerlat measurement without any load modules:

    sudo rteval -d 600 --measurement-module timerlat --noload

With Logging and Annotation

Run with logging enabled and add an annotation to the report:

    sudo rteval -d 3600 -L -a "Testing kernel 6.17.7"

CPU Isolation with Cpusets

Run with cpusets to isolate measurement on cores 0-3, loads on cores 4-15, and reserve core 0 for housekeeping:

    sudo rteval --cpusets --measurement-cpulist 1-3 \
           --loads-cpulist 4-15 --housekeeping 0

Custom Timerlat Configuration

Run timerlat with custom interval and stop trace threshold:

    sudo rteval -d 1800 --measurement-module timerlat \
           --timerlat-interval 1000 --timerlat-stoptrace 100

Analyze Existing Reports

Summarize multiple existing XML reports:

    rteval -Z rteval-20260601-1/summary.xml \
              rteval-20260602-1/summary.xml

Files

rteval-YYYYMMDD-N/

Report directory created for each run, containing:

summary.xml: XML report with system info, measurements, and statistics
logs/: Directory containing measurement and load module logs (with -L option)

Measurement Modules

cyclictest: Measures scheduling latency by running high-priority SCHED_FIFO threads that sleep for a specified interval and measure wakeup latency. This is the traditional measurement tool for realtime Linux testing.
timerlat: Uses the rtla (Real-Time Linux Analysis) tool's timerlat tracer to measure timer latency. This is a more modern approach that leverages kernel tracing infrastructure and can provide more detailed analysis of latency sources.

Load Modules

kcompile: Performs parallel kernel compilation to create CPU and I/O load. Compiles with allmodconfig to maximize compilation work.
hackbench: Scheduler stress test that creates pairs of threads exchanging data via pipes or sockets, exercising the kernel scheduler.
stress-ng: Flexible stress testing tool with many stressor options for CPU, memory, I/O, and more.

Notes

Root privileges are required to run rteval as it needs to set real-time scheduling priorities and access system resources.

For production testing, run for at least 12-24 hours to capture rare latency events. The default 60-second duration is only suitable for quick validation.

When using cpusets (--cpusets), ensure your kernel has cgroup v2 support and the cpuset controller is available.

The timerlat measurement module requires a kernel with rtla support (kernel 5.15+ with CONFIG_OSNOISE_TRACER enabled).