Changing the default MTU on a MacOS Docker Desktop to allow Linking XDP eBPF programs

Docker_Engine_Config.json

{
  "mtu": 1500
}

Could not attach XDP program: create link: numerical result out of range

The error from the title occurred while trying to link an ebpf program to the kernels XDP on my MacOS from inside a docker container.

Here is the source code - https://github.com/cilium/ebpf/blob/main/examples/xdp/main.go
It's from ciliums example repository.

Running The eBPF Program

I have this clang file this I want to link onto my kernel:

SEC("xdp")
int xdp_prog_func(struct xdp_md *ctx) {
	__u32 ip;
	__u8 protocol;
	if (!parse_ip_src_addr(ctx, &ip, &protocol)) {
		// Not an IPv4 packet, so don't count it.
		goto done;
	}

	__u32 *pkt_count = bpf_map_lookup_elem(&xdp_stats_map, &protocol);
	if (!pkt_count) {
		// No entry in the map for this IP address yet, so set the initial value to 1.
		__u32 init_pkt_count = 1;
		bpf_map_update_elem(&xdp_stats_map, &protocol, &init_pkt_count, BPF_ANY);
	} else {
		// Entry already exists for this IP address,
		// so increment it atomically using an LLVM built-in.
		__sync_fetch_and_add(pkt_count, 1);
	}

done:
	// Try changing this to XDP_DROP and see what happens!
	return XDP_PASS;
}

This should count how many packets each protocol had up to this moment. To compile it we use go generate. It then generates some .o and .go files that our Golang program can use.

The following Golang file loads the clang onto the kernel:

// Attach the program.
l, err := link.AttachXDP(link.XDPOptions{
	Program:   objs.XdpProgFunc,
	Interface: iface.Index,
})
if err != nil {
	log.Fatalf("could not attach XDP program: %s", err)
}
defer l.Close()

By running go build we create the executable, that we will run to run our ebpf program.

The Problem

When I tried running the executable with the network interface eth0 I received the following error:

could not attach XDP program: create link: numerical result out of range

Me and my co-founders thought it's connected to the network interface eth0 but did not have a clue on how to check it.

Maybe MacOS has problems with eth0 and en0 or something similar as using lo instead of eth0 worked as planned.

Fix

After chatting with 2 nice guys on ebpfs slack - here is a link to the full discussion https://cilium.slack.com/archives/C4XCTGYEM/p1710150479896969

We understood that's the problem is the MTU on my host machine (my new MacBook 14 Pro M2 Pro), the MacOSs default MTU value is too big:

docker network inspect f570862f920d | rg com.docker.network.driver.mtu
            "com.docker.network.driver.mtu": "65535"

I had to change the default value for com.docker.network.driver.mtu from 65535 to 1500:

Now it looks like this:

"com.docker.network.driver.mtu": "1500"

And I can count the packets:

> go generate && go build && sudo ./ebpf-xdp eth0
Compiled /ebpf/bpf_bpfel.o
Stripped /ebpf/bpf_bpfel.o
Wrote /ebpf/bpf_bpfel.go
Compiled /ebpf/bpf_bpfeb.o
Stripped /ebpf/bpf_bpfeb.o
Wrote /ebpf/bpf_bpfeb.go
2024/03/11 12:04:04 The chosen interface is eth0 with index 16
2024/03/11 12:04:04 Attached XDP program to iface "eth0" (index 16)
2024/03/11 12:04:04 Press Ctrl-C to exit and remove the program
2024/03/11 12:04:39 Map contents:
        6 => 6
        17 => 2
2024/03/11 12:04:44 Map contents:
        6 => 6
        17 => 2
2024/03/11 12:04:49 Map contents:
        6 => 6
        17 => 2
2024/03/11 12:04:54 Map contents:
        6 => 12
        17 => 4
2024/03/11 12:04:59 Map contents:
        6 => 32
        17 => 12
2024/03/11 12:05:04 Map contents:
        6 => 36
        17 => 12

References

• https://docs.docker.com/reference/cli/dockerd/#daemon-configuration-file
• https://cilium.slack.com/archives/C4XCTGYEM/p1710150479896969