Skip to content

Instantly share code, notes, and snippets.

@hitech95

hitech95/doc_udp_loss_egress.md

Last active January 19, 2026 12:56
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save hitech95/2a8e00030e2e21b4bbee69e11d27e589 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save hitech95/2a8e00030e2e21b4bbee69e11d27e589 to your computer and use it in GitHub Desktop.
Download ZIP
UDP Packet Loss Investigation, SKY WIFI
Raw
doc_udp_loss_egress.md

UDP Packet Loss / Malformed Packet Investigation

1. Scope and Context

This document describes the investigation of a UDP packet loss / corruption issue in a virtualized networking environment.

While iperf is used as a controlled and repeatable test case, the issue is not limited to iperf. Similar behavior has been observed in other UDP-based applications. iperf is used solely to demonstrate and quantify the issue.

2. Problem Description

  • UDP packets sent by the server (this system) are:
    • Dropped
    • Not received
    • Received malformed
  • The issue:
    • Is directional (server → client)
    • Increases with throughput
    • Is reproducible
  • TCP traffic is not affected
  • The issue occurs under multiple conditions, not limited to iperf testing

3. Test Methodology (iperf as Reference)

3.1 Tool

  • iperf (UDP mode, TCP mode just to confirm no other issues are present on the link)

3.2 Roles

  • Server: This environment (tested via LXC container, Docker container, and dedicated VM to eliminate container-specific issues)
  • Clients: External and internal test hosts

3.3 Test Parameters

Parameter Values
Direction Normal, Reverse (-R)
Bandwidth 1M, 5M, 10M, 15M (--bitrate)
Protocol UDP (--udp)

3.4 Results Summary

Bandwidth Normal Reverse (-R)
1M OK Mostly OK
5M OK Mostly OK
10M OK Some loss
15M OK Consistent and significant loss

4. Network Architecture Overview

The network consists of an FTTH fiber connection, an ONT, a Proxmox virtualization host, and an OpenWrt router VM with multiple NICs including PCIe passthrough for WAN and LACP LAN aggregation. The server runs inside LXC, Docker, or a dedicated VM, and clients connect either via WAN (Client A) or LAN/subnet cross (Client B). Bridges (vmbr0/vmbr1) are used for container/VM connectivity.

5. Router VM Configuration

  • 4 physical NICs (PCIe passthrough)
    • eth2 → WAN
    • Remaining 3 NICs → aggregated into LACP0 for LAN
  • eth0 → virtio device, part of br-lan bridge
  • br-lan bridge composed of eth0 + LACP0
  • LACP0 VLAN-aware but no VLANs configured
  • Subnets & firewall zones:
    • br-lan → 192.168.1.0/24 → lan zone
    • public → 192.168.100.0/24 → public zone
    • wan → ISP-assigned → wan zone
  • Cross-subnet traffic requires explicit firewall rules and may traverse DNAT when moving between zones

6. Proxmox Host & Server Layout

  • vmbr0 → Linux bridge, no IP
  • vmbr1 → Linux bridge, no IP
  • LXC / Docker containers or dedicated VMs can attach to either bridge
  • Bridges act purely at L2
  • PVE host physical NIC also connected to main switch (for some tests)

7. Test Clients

Client Connection Notes
Client A External (WAN) Cross-ISP testing over FTTH or 4G
Client B Internal LAN Connected to main switch; traffic crosses subnets and zones, traversing firewall and DNAT rules

8. Detailed ASCII Network Diagram

        +--------------+
        |   Client A   |
        |   (WAN/ISP)  |
        +--------------+
                |
              Fiber
                |
              [ONT]
                |
          Ethernet RJ45
                |
             eth2 (WAN)
                |
        +--------------------------+
        |     PVE Host             |
        |--------------------------|
        | vmbr0 (bridge)           |
        | vmbr1 (bridge)           |
        |   |                      |
        |   +--> LXC Container     |
        |   |       UDP Server     |
        |   |       192.168.100.X  |
        |   +--> Docker / VM test  |  (alternative to LXC)
        |--------------------------|
        | OpenWrt Router VM        |
        |--------------------------|
        | eth2 -> WAN              | (NIC PCIe passthrough)
        |                          |
        | eth0 (virtio) -> br-lan  | (mapped to vmbr0)
        | lacp0 -> eth3-5          | (LACP uses 3 NICS in PCIe passthrough)
        | br-lan -> eth0 + LACP0   |
        | [LAN] 192.168.1.0/24     | 
        |                          |
        | eth1 (virtio) -> public  | (mapped to vmbr1)
        | [PUBLIC] 192.168.100.0/24|
        |                          |
        | Firewall / NAT / Zones   |
        +--------------------------+
                |       (lacp0)
                +--------------------> Switch -> Client B
                |       (eth0)
                +--------------------> vmbr0 -> Server VM/Container -> Client B

9. Performed Tests and Results

Client Network Location ISP / Access Type Result
Client A External (WAN) TIM – FTTH ❌ Packet loss present
Client A External (WAN) TIM – 4G ❌ Packet loss present
Client A External (WAN) Vodafone – 4G ❌ Packet loss present
Client A External (WAN) Wind3 – 4G ✅ No packet loss
Client B Internal LAN N/A ✅ No packet loss (traffic crosses subnets, firewall, and DNAT rules)

10. Server Placement Isolation Tests

Server Location Network Attachment Result
LXC container vmbr1 (virtio) ❌ Issue present
Docker container on PVE host vmbr1 (virtio) ❌ Issue present
Dedicated VM vmbr1 (virtio) ❌ Issue present
PVE host (bare metal) Physical NIC to main switch ✅ No issue
OpenWrt router VM Local router interface ✅ No issue

11. Path Analysis

Failing Path (Internet clients → server)

Client A (Internet)
  |
  v
WAN (ISP)
  |
  v
eth2 (PCIe passthrough, WAN)
  |
  v
OpenWrt Firewall / NAT
  |
  v
OpenWrt eth1 (virtio) -> public subnet 192.168.100.0/24
  |
  v
vmbr1 (PVE bridge)
  |
  v
Server (LXC / Docker / dedicated VM)

Packet loss is observed on this path when the OpenWrt VM uses virtio NICs, regardless of whether the server is attached to vmbr1 or vmbr0.

Working Paths (Internal-only clients)

Client A (Internet)
  |
  v
WAN (ISP)
  |
  v
eth2 (PCIe passthrough, WAN)
  |
  v
OpenWrt Firewall / NAT
  |
  v
OpenWrt br-lan (bridge) -> lan subnet 192.168.1.0/24
  |
  v
lacp0 
  |
  v
Switch
  |
  v
Server

The diagrams above show the direction of the connection from the Client to Server, but traffic would flow backward by using the -R option of iperf.

Internal-only traffic does not exhibit packet loss. Also in the case of the server behind vmbr1 and client behind vmbr0 cross subnets and firewall zones.

12. Observations and Implications

  • All failing cases involve Internet-based clients; purely internal paths work reliably.
  • The server may be connected to either vmbr1 or vmbr0; the choice of bridge does not change the outcome.
  • Packet loss is observed only when traffic enters from the WAN and is forwarded by the OpenWrt VM using virtio NICs.
  • The issue remains directional (server → client) and rate-dependent, affecting high-throughput UDP.
  • Proxmox bridges (vmbr0 / vmbr1) function correctly and are not a source of packet loss.
  • LXC, Docker, and dedicated VM server runtimes are ruled out.
  • If the iperf server is connected via the LACP path through the physical switch (i.e. not using vmbr interfaces), Internet-sourced traffic works correctly.
  • During the test the server has also been run in Openwrt VM binded to both lan and public subnets no issue is shown.
  • Traffic between LACP0 (or a VM/container attached to vmbr0) to the server that traverses routing/firewall logic and exits via vmbr1 also works correctly.
  • Internet ingress doesn't show any significant packet loss.
  • The failure therefore requires the combination of Internet egress + virtio forwarding toward a vmbr-attached server
  • Replacing OpenWrt VM virtio NICs with Intel E1000e fully mitigates the issue for all tested paths.
  • This strongly points to a defect or limitation in virtio-based forwarding inside OpenWrt under high UDP load, not in the host bridges or container stack.

13. TLDR

  • The problem manifests only for Internet-destinated traffic.
  • Server-side attachment to vmbr0 or vmbr1 does not influence the issue
  • Internal-only routing and firewalling paths are stable.
  • The issue is eliminated by switching OpenWrt VM NICs from virtio to Intel E1000e.
  • Most likely root cause: virtio NIC handling in OpenWrt when sending high-rate UDP traffic from virtual bridges to WAN.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment