When you need eBPF
eBPF lets you run small verified programs inside the kernel and collect data about events: process launches, file opens, disk latency, TCP connections. It helps when top, iostat and the logs are silent: the tool shows who is doing what right now, without changing application code. It sees kernel events, not the lines of your code.
bcc and bpftrace: two sets of tools
bcc is a set of ready-made utilities: install the package and run, say, opensnoop. bpftrace is a language for describing a trace in a few lines: more flexible, but you write the query yourself. Six tools exist in bcc: execsnoop, opensnoop, biolatency, tcpconnect, runqlat and profile. In bpftrace five are ready-made; profile is missing from its tools directory.
| Question | bcc | bpftrace, tools directory |
|---|---|---|
| Who starts processes | execsnoop | execsnoop.bt |
| Which files are opened | opensnoop | opensnoop.bt |
| Block I/O latency | biolatency | biolatency.bt |
| Outgoing TCP connections | tcpconnect | tcpconnect.bt |
| CPU scheduler queue | runqlat | runqlat.bt |
| Where the CPU time goes | profile | no ready-made file |
The list was checked against the bcc README and the tools directory of the bpftrace repository.
Kernel and privilege requirements
You need a recent enough kernel, root rights and, for some tools, kernel headers or a BTF file. The policy of current bpftrace releases is a minimum kernel of 6.1, and bcc requires a kernel no older than 4.1 plus kernel headers. The package in your distribution may be older than that policy, so check bpftrace --version.
| What | Value | Source |
|---|---|---|
| bpftrace, current releases | kernel 6.1 or newer | the bpftrace project policy |
| bcc | kernel 4.1 or newer, kernel headers | INSTALL.md of the bcc project |
| The file /sys/kernel/btf/vmlinux | available from kernel 5.5 | libbpf-tools README in bcc |
| CAP_BPF and CAP_PERFMON | added in kernel 5.8, before that CAP_SYS_ADMIN is needed | man capabilities |
For BTF it is more reliable not to memorise a version but to check whether the file exists: ls /sys/kernel/btf/vmlinux.
Installing on Ubuntu
Installation takes two commands. bcc comes as the bpfcc-tools package with the kernel headers, bpftrace as the package of the same name. On Ubuntu the bcc utilities live in /sbin with a -bpfcc suffix, such as opensnoop-bpfcc. bcc warns that Ubuntu packages go stale. On Debian the package names may match; confirm with apt search.
- Check the kernel version and whether the BTF file exists.
- Install the set you need from the block below.
- Run the utility as root for a few seconds and stop it with Ctrl+C.
uname -r
ls /sys/kernel/btf/vmlinux
sudo apt install bpfcc-tools linux-headers-$(uname -r)
sudo apt install bpftrace
sudo opensnoop-bpfccWhere it works: VPS, container, bare metal
On a KVM VPS the guest has its own kernel, so conditions match bare metal, but the image’s kernel may be stripped down: check BTF and the version. In an unprivileged container (LXC, Incus, Docker) eBPF usually fails with “Operation not permitted”. That is an observation, not a prohibition from the documentation, so check for yourself.
systemd-detect-virt
systemd-detect-virt --container
sudo opensnoop-bpfccThe systemd-detect-virt command prints the virtualisation type, for example kvm or lxc, and with the --container key the container type or none. If opensnoop-bpfcc printed lines within a few seconds, the environment is suitable; if you got “Operation not permitted”, eBPF is closed here.
Warning. On a shared kernel tracing sees the neighbours’ processes: it shows program launches and file opens across the whole system, not only in your container. That is why the tool is dangerous and needs root, and in shared environments it is usually locked down.
Four ready-made examples
Start with short runs of a few seconds: each example answers its own question, and all of them stop with Ctrl+C. The first three use ready-made bcc utilities, the fourth a bpftrace one-liner. The bcc commands are given with the -bpfcc suffix, as on Ubuntu; on other distributions the name may have no suffix.
execsnoop-bpfcc— who starts processes. It catches short-lived programs that never reach top: needless script calls from cron or from a web application.opensnoop-bpfcc— which files are opened and by whom. It helps to find attempts to open a missing file and unexpectedly heavy disk access.biolatency-bpfcc 10 1— a histogram of block I/O latency over 10 seconds. A long tail means the disk or its queue is slow.- a bpftrace one-liner with the
profile:hz:99probe — counts which processes the CPU most often ends up in, 99 times per second; there is no ready-made profile in bpftrace.
sudo execsnoop-bpfcc
sudo opensnoop-bpfcc
sudo biolatency-bpfcc 10 1
sudo bpftrace -e 'profile:hz:99 { @[comm] = count(); }'What to do with the result
An eBPF result is a hint, not a diagnosis: it names a process, a file or a device, and an ordinary investigation has to follow. For the overall picture, Linux performance profiling will do, and a disk latency histogram leads to the article on the disk subsystem and queues.
- Slow PHP scripts: the PHP-FPM slowlog shows the line of code that eBPF cannot see.
- Traces of the same events in the journal: journalctl and rsyslog.
Frequently asked questions
Is it safe to run eBPF on a production server?
Short runs are usually tolerated, but tracing frequent events adds load. Start with narrow tools such as biolatency, run them for a few seconds and do not leave tracing switched on.
Why does bpftrace print “Operation not permitted”?
Most often you lack root rights or you are inside an unprivileged container where BPF access is closed. Check systemd-detect-virt --container and run the command through sudo on the server itself or on a KVM VPS.
What should ZevsHost customers do?
On shared hosting such diagnostics are unavailable: the kernel is shared, and the panel shows load and logs. On a VPS with KVM virtualisation the kernel is your own and you run the tools yourself; the virtualisation type is shown by systemd-detect-virt.