For many symbolic probing operations, systemtap needs DWARF debuginfo for the relevant binaries. This often includes resolving function/statement probes, or $context variables in related handlers. DWARF debuginfo is created by the compiler when using CFLAGS -g, and may be found in the original binaries built during compilation, or may have been split into separate files. The SYSTEMTAP_DEBUGINFO_PATH environment variable affects where systemtap looks for these files.
If your operating system came from a distributor, check with them if debuginfo packages or variants are available. If your distributor does not have debuginfo-equipped binaries at all, you may need to rebuild it.
Systemtap uses the elfutils library to process ELF/DWARF files. The version of elfutils used by systemtap is the number after the slash in the -V output:
% stap -V Systemtap translator/driver (version 4.2/0.178, rpm 4.2-1.fc30) Copyright (C) 2005-2019 Red Hat, Inc. and others [...]
This indicates systemtap version 4.2 with elfutils version 0.178.
New enough versions of elfutils (0.178+) enable systemtap to automatically download correct debuginfo from servers run by you, your organization, and/or someone on the public internet. Try:
% export DEBUGINFOD_URLS=https://debuginfod.elfutils.org/ % export DEBUGINFOD_PROGRESS=1 # for progress messages, if you like
and rerun systemtap. It might just work. If it doesn't, read on.
- kernel debuginfo
For scripts that target the kernel, systemtap may search for the vmlinux file created during its original build. This is distinct from the boot-loader's compressed/stripped vmlinuz file, and much larger. If you have a hand-built kernel, make sure it was built with the CONFIG_DEBUG_INFO=y option. Some Linux distributions may include several kernel variants, including a confusingly named kernel-debug (an alternative kernel, with its own kernel-debug-debuginfo package), which is not the same thing as the kernel-debuginfo (DWARF data for the base kernel). The stap-prep program can help install the right set.
- process debuginfo
For scripts that target user-space, systemtap may search for debuginfo. If you have hand-built binaries, use CFLAGS=-g -O2 to compile them.
On some systems, binaries may be compiled with a subset of debuginfo useful for function tracing and backtraces. This 'Minidebuginfo' is a xz compressed section labeled .gnu_debugdata. Support for minidebuginfo relies on elfutils version 0.156 or later.
- compressed debuginfo
On some systems, debuginfo may be available, but compressed into .zdebug_* sections. Support for compressed debuginfo relies on elfutils version 0.153 or later.
- unnecessary debuginfo
In some cases, a script may be altered to avoid requiring debuginfo. For example, as script that uses probe syscall.* probes could try instead probe nd_syscall.* (for non-DWARF syscall): these work similarly, and use more intricate (fragile) tapset functions to extract system call arguments. Another option is use of compiled-in instrumentation such as kernel tracepoints or user-space <sys/sdt.h> markers in libraries or executables, which do not require debuginfo. If debuginfo was required for resolving a complicated $var->foo->bar expression, it may be possible to use @cast(var,"foo","foo.h")->foo->bar to synthesize debuginfo for that type from a header file.
On some platforms, systemtap may advise what commands to run, in order to download needed debuginfo. Another possibility is to invoke systemtap with the --download-debuginfo flag, which uses ABRT. The stap-prep script included with systemtap may be able to download the appropriate kernel debuginfo. Another possibility is to install and use a stap-server remote-compilation instance on a machine on your network, where debuginfo and compilation resources can be centralized. Try the stap --use-server option, in case such a server is already running.
gcc(1), stap(1), stappaths(7), stap-server(8), stap-prep(1), strip(1), warning::symbols(7stap), error::dwarf(7stap), error::reporting(7stap), error::contextvars(7stap), debuginfod(8), http://elfutils.org/, https://sourceware.org/elfutils/Debuginfod.html, http://fedoraproject.org/wiki/Features/MiniDebugInfo