berglh/minimal-zfs-root.md

## minimal-zfs-root.md

      
    Raw
  

              minimal-zfs-root.md
            
          
Minimal ZFS Root on Ubuntu 25.04


Topics


Introduction
Prepare environment

Partitions
Create Zpools


Create ZFS File Systems

Create ZFS Datasets
Bootstrap Ubuntu


System Configuration

Chroot into installation


GRUB Installation
First Boot
Full Software Installation
Final Cleanup
Extras
Recovery

Introduction

This follows the guide Ubuntu 22.04 Root on ZFS with some changes.

Custom structure of boot and root ZFS pools
Lack of intermediary containers i.e. ubuntu/root instead of rpool/ROOT/ubuntu.
Changes to work with Ubuntu 25.04 specifically

⚠️ Warning: If more than one Linux distribution will be used on the system, using the structure in the guide with intermediary containers is useful. It allows having different boot and root pools on the same physical device and booting to different operating systems using grub.
Prepare environment

Boot from the Ubuntu Live USB (ISO), preferably the same version you wish to install.
Update and install basic tools, installation via SSH is a bit easier so configure if required.
sudo apt update
apt install --yes openssh-server vim
Disable automounting
gsettings set org.gnome.desktop.media-handling automount false
Install ZFS in the Live CD environment:
apt install --yes debootstrap gdisk zfsutils-linux
systemctl stop zed
 
Disk Formatting

Set a variable with the disk name:
DISK=/dev/disk/by-id/nvme-Force_MP600_1932822900012855046A
Ensure swap parts are not in use:
swapoff --all
 
Destroy disk

If contained ZFS:
wipefs -a $DISK
If flash-storage:
blkdiscard -f $DISK
Clear partition table:
sgdisk --zap-all $DISK
Partitions

Create the partition layout for EFI, no BIOS partition required for EFI only.

This creates an EFI partition
This creates a boot partition as ZFS (which EFI is mounted in)
Adjust the swap partition size (-16G) to the desired size at the end of the disk
Puts swap partition at the end of the disk formatted as swap
Check sgdisk -p /dev/disk/by-id/nvme-Force_MP600_1932822900012855046A to ensure there is no sector overlaps

sgdisk -n1:0:+512M -t1:EF00 -c1:"EFI" $DISK
sgdisk -n2:0:+2G   -t2:BF00 -c2:"ZFS boot" $DISK
sgdisk -n3:0:-16G  -t3:BF00 -c3:"ZFS root" $DISK
sgdisk -n4:0:0     -t4:8200 -c4:"Linux Swap" $DISK
sgdisk -p $DISK
 
Format Swap

mkswap /dev/disk/by-id/${DISK}$-part4
 
Format EFI

apt install --yes dosfstools
mkdosfs -F 32 -s 1 -n EFI ${DISK}-part1
Create Zpools

 
Boot Pool

# boot pool
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o cachefile=/etc/zfs/zpool.cache \
    -o compatibility=grub2 \
    -o feature@livelist=enabled \
    -o feature@zpool_checkpoint=enabled \
    -O devices=off \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/boot -R /mnt \
    boot ${DISK}-part2
 
Root/OS Pool

# root (ubuntu) pool
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/ -R /mnt \
    ubuntu ${DISK}-part3
Create ZFS File Systems

zfs create -o mountpoint=/ \
    -o com.ubuntu.zsys:bootfs=yes \
    -o com.ubuntu.zsys:last-used=$(date +%s) ubuntu/root
zfs create -o mountpoint=/boot boot/strap
Create ZFS Datasets

Create home container & datasets.
zfs create -o canmount=off -o mountpoint=/ ubuntu/home
zfs create -o canmount=on -o mountpoint=/root ubuntu/home/root
chmod 700 /mnt/root

# zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off ubuntu/usr
zfs create -o com.ubuntu.zsys:bootfs=no -o canmount=off ubuntu/var
zfs create ubuntu/var/lib
zfs create ubuntu/var/log
zfs create ubuntu/var/spool
zfs create ubuntu/var/cache
zfs create ubuntu/var/mail
Create tmp dataset allows excluding from snapshots
zfs create -o com.ubuntu.zsys:bootfs=no ubuntu/tmp
chmod 1777 /mnt/tmp
Mount a tmpfs at /run:
mkdir /mnt/run
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock
Bootstrap Ubuntu

#apt install --yes debootstrap
debootstrap plucky /mnt
Copy zpool.cache
mkdir /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/
System Configuration

hostname
echo quark > /mnt/etc/hostname
vi /mnt/etc/hosts # add 127.0.0.1 ${hostname}

Configure networking
ip a # get inteface name
vim /mnt/etc/netplan/01-network.yaml # add basic config

Change eno1 to your interface name. Configure static networking if required.
network:
  version: 2
  ethernets:
    eno1:
      dhcp4: true
Add app sources
ls /mnt/etc/apt/sources.list.d/
vim /mnt/etc/apt/sources.list.d/ubuntu.sources

Types: deb deb-src
URIs: http://au.archive.ubuntu.com/ubuntu/
Suites: plucky plucky-updates plucky-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

Types: deb deb-src
URIs: http://security.ubuntu.com/ubuntu
Suites: plucky-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Check keyring exists
ls /usr/share/keyrings/ubuntu-archive-keyring.gpg

Copy terminfo optional (for kitty)
cp -r ~/.terminfo/ /mnt/root/

Chroot into installation

mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login
Configure basic environment
apt update
Configure language and tzdata
locale-gen --purge "en_AU.UTF-8"
update-locale LANG=en_AU.UTF-8 LANGUAGE=en_AU
ln -fs /usr/share/zoneinfo/Australia/Brisbane /etc/localtime
dpkg-reconfigure -f noninteractive tzdata
Optional if you require a different keyboard layout.
dpkg-reconfigure console-setup
Install neovim or the preferred text editor
apt install --yes neovim
EFI File-system
apt install --yes dosfstools
mkdir /boot/efi
echo '# EFI' > /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part1) /boot/efi vfat umask=0077 0 1 >> /etc/fstab
mount /boot/efi
Put /boot/grub on the EFI System Partition
Note: For a single-disk install only.
mkdir /boot/efi/grub /boot/grub
echo /boot/efi/grub /boot/grub none defaults,bind 0 0 >> /etc/fstab
mount /boot/grub
Install GRUB/Linux/ZFS in the chroot environment for the new system (zsys is the guide, but doesn't work in 25.04)
apt install --yes grub-efi-amd64 grub-efi-amd64-signed linux-image-generic shim-signed zfs-initramfs
Optional: Remove os-prober to avoids error messages from update-grub. os-prober is only necessary in dual-boot configurations
apt purge --yes os-prober
Set root password
passwd
Configure swap for an unencrypted single-disk install
echo '# Swap' >> /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part4) none swap discard 0 0 >> /etc/fstab
swapon -a
Add default system groups. Include (lxd) if you plan on using system level containers.
addgroup --system lpadmin
addgroup --system sambashare
Install and configure openssh-server
apt install --yes openssh-server
GRUB Installation

Verify that the ZFS boot filesystem is recognised
grub-probe /boot
Refresh the initrd files
update-initramfs -c -k all
Edit grub config, some settings can be reverted later if preferred
vim /etc/default/grub

# Disable memory zoning: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1862822
# Add init_on_alloc=0 to: GRUB_CMDLINE_LINUX_DEFAULT

# Comment out: GRUB_TIMEOUT_STYLE=hidden
# Set: GRUB_TIMEOUT=5
# Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5
# Remove quiet and splash from: GRUB_CMDLINE_LINUX_DEFAULT
# Uncomment: GRUB_TERMINAL=console
# Save and quit.
update-grub
Update the boot configuration
update-grub
For UEFI booting, install GRUB to the ESP
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck --no-floppy
Fix filesystem mount ordering by activating zfs-mount-generator. This makes systemd aware of the separate mountpoints, which is important for things like /var/log and /var/tmp. In turn, rsyslog.service depends on var-log.mount by way of local-fs.target and services using the PrivateTmp feature of systemd automatically use After=var-tmp.mount.
mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/boot
touch /etc/zfs/zfs-list.cache/ubuntu
zed -F &
Verify that zed updated the cache by making sure these are not empty.
cat /etc/zfs/zfs-list.cache/boot
cat /etc/zfs/zfs-list.cache/ubuntu
If either is empty, force a cache update and check again:
zfs set canmount=on boot/strap
zfs set canmount=on ubuntu/root
If they are still empty, stop zed (as below), start zed (as above) and try again.
Once the files have data, stop zed:
fg
Press Ctrl-C.
Fix the paths to eliminate /mnt prefix.
sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*
Exit from the chroot environment back to the LiveCD environment:
exit
Run these commands in the LiveCD environment to unmount all filesystems
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a
If export failed due to ubuntu busy error, try to kill everything that might be using it:
zfs umount ubuntu/root
# Check if processes have the pool open
grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq
# Kill the processes
for p in $(grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq); do kill $p; done
zpool export -a
If even after that the pool is busy, mounting it on boot will fail. In the initramfs prompt type zpool import -f ubuntu, then exit in the initramfs prompt.
Reboot 🤞
reboot

Wait for the newly installed system to boot normally.
First Boot

Login as root
ssh root@server-address

📸 Optional: Take some snapshots
zfs snapshot boot/strap@install
zfs snapshot ubuntu/root@install

🤓 Create a user
username=berg
zfs create -o canmount=on -o mountpoint=/home/$username ubuntu/home/${username}
adduser ${username}
# This isn't necessary if adduser does it
cp -a /etc/skel/. /home/${username}
# optionally copy terminfo if using kitty etc
cp -r /root/.terminfo /home/${username}
# ensure all files are owned by the user
chown -R ${username}:${username} /home/${username}
usermod -a -G adm,cdrom,dip,lpadmin,lxd,plugdev,sambashare,sudo $username
Full Software Installation

Upgrade the minimal system
apt dist-upgrade --yes
Install a command-line environment only
apt install --yes ubuntu-standard
Install a full GUI environment
Hint: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager.
apt install --yes ubuntu-desktop
Disable hibernation if not required, will cause warnings in initramfs update
echo "RESUME=none" | sudo tee /etc/initramfs-tools/conf.d/resume
update-initramfs -u
Disable mkinitramfs not found messages
touch /etc/zfs/vdev_id.conf
touch /etc/zfs/initramfs-tools-load-key
mkdir -p /etc/zfs/initramfs-tools-load-key.d
touch /etc/zfs/initramfs-tools-load-key.d/_placeholder
update-initramfs -u
Disable log compression
As /var/log is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if using snapshots of /var/log, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in /etc/logrotate.d by hand to comment out compress, or use this loop (copy-and-paste highly recommended):
for file in /etc/logrotate.d/* ; do
    if grep -Eq "(^|[^#y])compress" "$file" ; then
        sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
    fi
done
Final Cleanup

Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally.
Optional: Disable the root password
sudo usermod -p '*' root
Add an SSH authorised key to the created user, that should be currently logged in.
mkdir ~/.ssh
# paste a public SSH key into authorized_keys
vim .ssh/authorized_keys
# set correct permissions for .ssh
# if these are not set, SSH key login will fail
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R $(whoami):$(whoami) ~/.ssh
⚠️ Important: Disconnect and try login with authorized_key before disabling the insecure login methods. If it is configured incorrectly and password login is disabled, login to the system via SSH will fail.
Disable root SSH logins. If you installed SSH earlier, revert the temporary change:
sudo vi /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
sudo systemctl restart ssh
Extras

Install landscape, shows system information on SSH login
sudo apt install landscape-common
Add terminal colours over SSH (scp ~/.bashrc from a Ubuntu desktop machine to home folder)
scp ~/.bashrc user@server:~
# then on the server
vim .bash_profile
Enter the following contents in the ~\.bash_profile file.
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi
Recovery

To recover the system from a Live USB ISO. After loading into the terminal from the Live USB installer.

Once in the termina, go through Prepare enviornment steps.
Mount everything correctly:
zpool export -a
zpool import -N -R /mnt ubuntu
zpool import -N -R /mnt boot
zfs mount ubuntu/root
zfs mount boot/strap
zfs mount -a

If needed, chroot into the installed environment.
mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock
chroot /mnt /bin/bash --login
mount -a

Perform recovery reparations
Cleanup by exiting from the chroot environment back to the LiveCD environment
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a

If export failed due to ubuntu busy error, try to kill everything that might be using it:
zfs umount ubuntu/root
# Check if processes have the pool open
grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq
# Kill the processes
for p in $(grep ubuntu/root /proc/*/mounts | cut -d/ -f3 | uniq); do kill $p; done
zpool export -a
No results found