bcc-zfsslower man page
zfsslower — Trace slow zfs file operations, with per-event details.
zfsslower [-h] [-j] [-p PID] [min_ms]
This tool traces common ZFS file operations: reads, writes, opens, and syncs. It measures the time spent in these operations, and prints details for each that exceeded a threshold.
WARNING: See the Overhead section.
By default, a minimum millisecond threshold of 10 is used. If a threshold of 0 is used, all events are printed (warning: verbose).
This uses kernel dynamic tracing of the ZPL interface (ZFS POSIX Layer), and will need updates to match any changes to this interface.
- This is intended to work with the ZFS on Linux project:
Since this uses BPF, only the root user can use this tool.
CONFIG_BPF and bcc.
-p PID Trace this PID only.
Minimum I/O latency (duration) to trace, in milliseconds. Default is 10 ms.
- Trace synchronous file reads and writes slower than 10 ms:
- Trace slower than 1 ms:
# zfsslower 1
- Trace slower than 1 ms, and output just the fields in parsable format (csv):
# zfsslower -j 1
- Trace all file reads and writes (warning: the output will be verbose):
# zfsslower 0
- Trace slower than 1 ms, for PID 181 only:
# zfsslower -p 181 1
Time of I/O completion since the first I/O seen, in seconds.
Type of operation. R == read, W == write, O == open, S == fsync.
File offset for the I/O, in Kbytes.
Size of I/O, in bytes.
Latency (duration) of I/O, measured from when it was issued by VFS to the filesystem, to when it completed. This time is inclusive of block device I/O, file system CPU cycles, file system locks, run queue latency, etc. It's a more accurate measure of the latency suffered by applications performing file system I/O, than to measure this down at the block device interface.
A cached kernel file name (comes from dentry->d_iname).
Completion timestamp, microseconds (-j only).
File offset, bytes (-j only).
Latency (duration) of the I/O, in microseconds (-j only).
This adds low-overhead instrumentation to these ZFS operations, including reads and writes from the file system cache. Such reads and writes can be very frequent (depending on the workload; eg, 1M/sec), at which point the overhead of this tool (even if it prints no "slower" events) can begin to become significant. Measure and quantify before use. If this continues to be a problem, consider switching to a tool that prints in-kernel summaries only.
Note that the overhead of this tool should be less than fileslower(8), as this tool targets zfs functions only, and not all file read/write paths (which can include socket I/O).
This is from bcc.
Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.
Unstable - in development.
biosnoop(8), funccount(8), fileslower(8)