#ebpf #bpf

facts.md

• Histogram bucket for negative values ("(..., 0)")
• https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md

tips.md

• Cheat Sheet
  • Probe Types
  • Variable Types
  • Builtin Variables
  • Builtin Functions
• Tips
  • Networking

Cheat Sheet

http://www.brendangregg.com/BPF/bpftrace-cheat-sheet.html

Probe Types

Alias	Type	Description
t	tracepoint	Kernel static instrumentation points
U	usdt	User-level statically defined tracing
k	kprobe	Kernel dynamic function instrumentation (standard)
kr	kretprobe	Kernel dynamic function return instrumentation (standard)
f	kfunc	Kernel dynamic function instrumentation (BPF based)
fr	kretfunc	Kernel dynamic function return instrumentation (BPF based)
u	uprobe	User-level dynamic function instrumentation
ur	uretprobe	User-level dynamic function return instrumentation
s	software	Kernel software-based events
h	hardware	Hardware counter-based instrumentation
w	watchpoint	Memory watchpoint events
p	profile	Timed sampling across all CPUs
i	interval	Timed reporting (from one CPU)
	iter	Iterator tracing over kernel objects
	BEGIN	Start of bpftrace
	END	End of bpftrace

Variable Types

Variable	Description
@name	global
@name[key]	hash
@name[tid]	thread-local
$name	scratch

Builtin Variables

Variable	Description
pid	Process ID
tid	Thread ID
uid	User ID
username	Username
comm	Process or command name
curtask	Current task_struct as a u64
nsecs	Current time in nanoseconds
elapsed	Time in nanoseconds since bpftrace start
kstack	Kernel stack trace
ustack	User-level stack trace
arg0...argN	Function arguments
args	Tracepoint arguments
retval	Function return value
func	Function name
probe	Full probe name
$1...$N	Positional parameters
cgroup	Default cgroup v2 ID

Builtin Functions

Variable	Description
pid	Process ID
tid	Thread ID
uid	User ID
username	Username
comm	Process or command name
curtask	Current task_struct as a u64
nsecs	Current time in nanoseconds
elapsed	Time in nanoseconds since bpftrace start
kstack	Kernel stack trace
ustack	User-level stack trace
arg0...argN	Function arguments
args	Tracepoint arguments
retval	Function return value
func	Function name
probe	Full probe name
$1...$N	Positional parameters
cgroup	Default cgroup v2 ID

Tips

• bpftrace one-liner syntax

bpftrace -e '<tracepoint> <filter> {
  <command>
}'

• To find the tracepoint's arguments

tplist-bpfcc  -v <tracepoint>
bpftrace -lv <tracepoint>

• Count open tracepoint's stacks for PID

stackcount-bpfcc -p <PID> '*'

# Example
stackcount-bpfcc -p 1 't:syscalls:sys_exit_exit'
Tracing 1 functions for "t:syscalls:sys_exit_exit"... Hit Ctrl-C to end.

Networking

• gethostlatency