XRDP + NVIDIA renderD129 (Arch)

xrdp-nvidia-arch.md

Arch + xRDP + RFX — working stack (no memory leak, correct colors)

Disclaimer

This gist was drafted with help from an AI assistant based on a live debugging session. Verify details before applying to your system.

TL;DR

Use xorgxrdp (non-glamor) with RFX codec. The glamor build (xorgxrdp-glamor) has a GEM shmem leak that causes OOM regardless of GPU or kernel version. Software rendering avoids the leak entirely with no visible performance penalty.

The problem

xorgxrdp-glamor uses GPU-accelerated rendering via DRI3/GEM. When xRDP negotiates the RFX codec, GEM shmem allocations grow continuously (~12 MB/s) and are never freed:

• Intel i915: shmem grows unbounded → OOM
• NVIDIA: fills VRAM → Failed to allocate NVKMS memory for GEM object → cascade crash

Tested on kernels 6.18.13-zen1-1-zen and 6.19.6-arch1-1 — the leak is not kernel-specific.

The fix

# Replace glamor with software-rendering xorgxrdp
sudo pacman -Rdd xlibre-xorgxrdp-glamor --noconfirm
sudo pacman -S xlibre-xorgxrdp --noconfirm
sudo systemctl restart xrdp xrdp-sesman

RFX encoding is CPU-based regardless of glamor, so there is no codec performance regression.

System specs

• Kernel: linux-zen (any version works — leak is not kernel-specific)
• GPUs: Intel HD 630 (renderD128) + NVIDIA GTX 1050 Mobile (renderD129)
• X server: XLibre X Server 1.25.0
• xRDP: 0.10.5
• xorgxrdp: xlibre-xorgxrdp 0.10.4-1 (non-glamor)

Packages

• xrdp 0.10.5-1
• xlibre-xorgxrdp 0.10.4-1 ← non-glamor, this is the key
• xlibre-xserver-common 25.0.0.21-1
• nvidia-580xx-dkms 580.126.18-2
• pipewire-module-xrdp 0.2-1

Codec config

/etc/xrdp/gfx.toml:

[codec]
order = [ "RFX", "H.264" ]

RFX gives correct colors. H.264 produces washed-out colors in Remmina because xRDP always encodes YUV 4:2:0 (the profile = "high444" setting is ignored — AVC444 encoding is not implemented in xRDP). Other clients like xfreerdp3 decode H.264 colors correctly.

GPU allocation (dual GPU system)

With non-glamor xorgxrdp, xRDP does not use any GPU. This frees both GPUs for other workloads:

Service	GPU	Device
xRDP (RFX)	None (software rendering)	—
Jellyfin (VAAPI transcoding)	Intel HD 630	/dev/dri/renderD128 + card1
NVIDIA GTX 1050	Free for other use	/dev/dri/renderD129 + card0

Browser memory optimization

For Chromium-based browsers over RDP, add to ~/.config/{brave,chrome,chromium}-flags.conf:

--disable-gpu
--process-per-site
--renderer-process-limit=4
--disk-cache-size=104857600
--disable-features=AcceleratedVideoDecodeLinuxGL,VaapiVideoDecodeLinuxGL

--disable-gpu is important — GPU compositing over RDP is wasteful and can cause rendering artifacts.

Verification

# Watch shmem — should stay stable, not grow continuously
watch -n5 'grep Shmem /proc/meminfo'

# Confirm RFX codec negotiated
sudo journalctl -u xrdp --no-pager | grep -i "matched"
# Should show: "Matched RFX mode"

Notes

• preset = "ultrafast" in gfx.toml forces Constrained Baseline profile in x264, overriding profile. Use superfast or slower.
• The reference postmarketOS server (xRDP 0.10.5, kernel 6.0) also uses non-glamor xorgxrdp without issues.
• Glamor is only needed for GPU-accelerated 2D rendering (OpenGL compositing). For RDP remote desktop this is unnecessary — the framebuffer is captured and encoded by the CPU regardless.