Quick and dirty k3s/metallb/longhorn setup

k3s.md

Overview

Quick and dirty tutorial for setting up a K3S cluster on three nodes with MetalLB and Longhorn.

• All commands are assumed to be run as root, within /root
• I barely know what I'm doing so if it works that's great!

With my setup;

• Three systems with a single NVMe drive.
• Each system has two 10Gb interfaces (bonded, LAGC).
• VLAN off that bond as the main interface.
• The built-in 1Gb interface is configured as a management/monitoring interface.

OS

Debian 12 minimal installation

Use LVM and only use ~100GB for the base install. Leave the rest for later expansion (Longhorn, for example)

Packages

apt-get update && apt-get install vim curl open-iscsi

Note that open-iscsi is only needed if you're installing Longhorn.

/etc/hosts

10.0.0.1 k3s1
10.0.0.2 k3s2
10.0.0.3 k3s3

SSH

On k3s1:

ssh-keygen
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
scp .ssh/{authorized_keys,id_rsa*} k3s2:.ssh/
scp .ssh/{authorized_keys,id_rsa*} k3s3:.ssh/

K3S

Init

On k3s1

openssl rand -hex 24 > secret
scp secret k3s2:
scp secret k3s3:

Ignore node-ip, bind-address and flannel-iface if your system has a single interface.

curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat secret) sh -s - server --cluster-init \
                                                                      --disable=servicelb \
                                                                      --node-ip=$(ip -o -4 a | awk -F'[\ \/]' '/vlan2/ {print $7}') \
                                                                      --bind-address=$(ip -o -4 a | awk -F'[\ \/]' '/vlan2/ {print $7}') \
                                                                      --flannel-iface='bond1.vlan2'

I have two interfaces; hence the need to specify.

Joining cluster

On k3s2 and k3s3

Ignore node-ip, bind-address and flannel-iface if your system has a single interface.

curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat secret) sh -s - server --server=https://10.0.0.1:6443 \
                                                                      --disable=servicelb \
                                                                      --node-ip=$(ip -o -4 a | awk -F'[\ \/]' '/vlan2/ {print $7}') \
                                                                      --bind-address=$(ip -o -4 a | awk -F'[\ \/]' '/vlan2/ {print $7}') \
                                                                      --flannel-iface='bond1.vlan2'

MetalLB

https://metallb.io/installation/#installation-by-manifest

Install

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml

Setup

Create a pool of addresses that won't overlap with your DHCP server:

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: k3s-metallb-pool1
  namespace: metallb-system
spec:
  addresses:
  - 10.0.0.4-10.0.0.9

I'll assume you want L2Advertisement, otherwise read the docs.

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: k3s-metallb-pool1
  namespace: metallb-system

Apply both those manifests

Longhorn

https://longhorn.io/docs/1.8.0/deploy/install/install-with-kubectl/

By default Longhorn stores data in /var/lib/longhorn. If you want that to be a separate mount point then set that up before installing Longhorn.

Default mount

This step needs to be run on each host.

Create volume group, and format it:

lvcreate -l90%FREE -n longhorn $(hostname)-vg
mkfs -t ext4 /dev/$(hostname)-vg/longhorn

Make and configure mount point:

mkdir /var/lib/longhorn
echo "/dev/$(hostname)-vg/longhorn /var/lib/longhorn ext4 defaults 0 2" >> /etc/fstab
systemctl daemon-reload
mount /var/lib/longhorn

Install

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.8.0/deploy/longhorn.yaml