bcc-softirqslower - Man Page

softirqslower.py [min_us]
softirqslower.py [-c CPU] [min_us]

The softirqslower tool traces softirq events that exceed a specified latency threshold. This tool helps diagnose interrupt handling bottlenecks by measuring two critical latency dimensions:

• irq(hard) to softirq: Time delay between hardware interrupt completion and softirq execution start
• softirq runtime: Actual execution duration of softirq handlers

Developed as part of the BCC (BPF Compiler Collection) tools, it's particularly useful for network stack optimization, scheduler tuning, and real-time systems analysis.

min_us

Minimum latency threshold to trace (in microseconds). Default: 10,000 μs (10 ms)

-c,  --cpu CPU

Trace only events on the specified CPU core

The tool outputs events in the following table format:

TIME     STAGE                SOFTIRQ  LAT(us)        CPU    COMM

With fields defined as:

TIME

Event timestamp (HH:MM:SS format)

STAGE

Latency measurement type:

irq(hard) to softirq

Time from hardware interrupt completion to softirq dispatch

softirq runtime

SoftIRQ handler execution duration

SOFTIRQ

SoftIRQ category (case-sensitive). Common values:

NET_RX

Network reception processing

NET_TX

Network transmission processing

TIMER

Timer callbacks

SCHED

Scheduler operations

RCU

Read-Copy-Update synchronization

TASKLET

Deferred task execution

HRTIMER

High-resolution timers

BLOCK

Block device operations

LAT(us)

Measured latency in microseconds

CPU

CPU core where softirq was handled (0-based numbering)

COMM

Process context handling the softirq:

swapper/N

Idle thread for CPU N

ksoftirqd/N

Softirq daemon for CPU N

<process>

User-space process

Trace softirqs exceeding 10μs latency:

softirqslower 10

Monitor only CPU core 1:

softirqslower -c 1

Trace network-related softirq delays (>50μs):

softirqslower 50 | grep -E 'net_rx|net_tx'

Capture RCU delays longer than 100μs to file:

softirqslower 100 | grep rcu > rcu_latency.log

Ctrl+C Stop tracing and exit gracefully

Chenyue Zhou

runqslower(8), hardirqs(8), softirqs(8), trace(8), funclatency(8),

BCC Tools Issue Tracker: https://github.com/iovisor/bcc/issues