ZFS pull backup with minimal permissions

README.md

Usage

on system that should be backed up

• put "restrict_commands.sh" in /usr/local/bin and make it executable
• install ts, lzop and optionally mbuffer

useradd zfsbackup --create-home --system
mkdir /home/zfsbackup/.ssh
zfs allow -u zfsbackup send,hold tank/dataset
echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys
chown zfsbackup:zfsbackup /home/zfsbackup/.ssh -R

on backup server

run cronjob with:

syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@target:tank/dataset tank/dataset

restrict_commands.sh

#!/bin/bash
export PATH=/usr/sbin:$PATH

_RE_DATASET=$'[\"\']+[a-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[a-z0-9/_]+[\"\']*@[\"\']*[a-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'

_WHITELIST=(
   -e "exit"
   -e "echo -n"
   -e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
   -e "zfs get -H syncoid:sync $_RE_DATASET"
   -e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
   -e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e "command -v (lzop|mbuffer)"
)

## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
    | ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
    | logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash

Sorry I'm kind of new to github, but I wanted to make a few suggestions

1. On the readme.md file
   please add mkdir /home/zfsbackup/.ssh
   else the echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys fails.
   

2. On the restrict_commands.sh:
   There is no mention where to place this file, I ended up creating a .local/bin folder in the zfsbackup user's home path
   mkdir -p /home/zfsbackup/.local/bin
   and adding that path in line 2 of the restrict_commands.sh so that it reads
   export PATH=$PATH:$HOME/.local/bin:/usr/sbin
   instead of export PATH=/usr/sbin:$PATH

3. Finally, if the restrict_commands.sh file fails with unknown command on line 26 error, it is because it is likely missing the ts command
   so in order to fix that I installed moreutils
   apt-get install moreutils

NOTE: just to be on the safe side it may be useful to ensure the user zfsbackup has permission to the folders we created on his home folder,
so to be safe run chown zfsbackup:zfsbackup /home/zfsbackup --recursive

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location? because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

Well no, the ssh key usage is correct. The backup job running on the backup server has to be run as root (it is not possible on linux to allow a non-root user to "zfs receive"). So root@backup has the private key, and logs into the server-to-be-backupped using the zfsbackup user account.

Thanks for your suggestions, I updated the readme accordingly. And I'll try to find out if running ssh on a different port will cause trouble.

any luck? i'm stucked on the lzop error also :(

syncoid --sshkey=/var/lib/syncoid/kubera-2022-05-31 backup@10.10.10.10:zpool/nixos/etc tank/test --no-privilege-elevation --debug --sendoptions="w c " --sshoption=StrictHostKeyChecking=off
DEBUG: SSHCMD: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31
loginShellInit
DEBUG: checking availability of lzop on source...
DEBUG: checking availability of lzop on target...
DEBUG: checking availability of lzop on local machine...
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source zpool/nixos/etc to target tank/test.
DEBUG: getting current value of syncoid:sync on zpool/nixos/etc...
ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -H syncoid:sync ''"'"'zpool/nixos/etc'"'"''
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'tank/test' 2>&1 |"...
DEBUG: getting list of snapshots on zpool/nixos/etc using ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'zpool/nixos/etc'"'"'' |...
DEBUG: creating sync snapshot using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs snapshot ''"'"'zpool/nixos/etc'"'"''@syncoid_kubera_2024-04-01:21:54:02-GMT02:00
"...
loginShellInit
DEBUG: target tank/test does not exist.  Finding oldest available snapshot on source zpool/nixos/etc ...
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs send -w -c  -nvP ''"'"'zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly'"'"'' 2>&1 |"...
DEBUG: sendsize = 782528
INFO: Sending oldest full snapshot zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly (~ 764 KB) to new target filesystem:
DEBUG: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test'
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
lzop: <stdin>: not a lzop file
mbuffer: error: outputThread: error writing to <stdout> at offset 0x10000: Broken pipe
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                 ]  0%
cannot receive: failed to read from stream
mbuffer: warning: error during output to <stdout>: Broken pipe
CRITICAL ERROR: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test' failed: 256 at /usr/sbin/syncoid line 492.

I also encountered the above mysterious lzop: <stdin>: not a lzop file error. I initially tried to work around it by specifying --compress=none, but that resulted in a different cannot receive: failed to read from stream. Turns out this is a bit of a red herring.

/var/log/syslog on the remote host (source of the dataset) revealed that it was catching multiple "non-whitelisted" commands. It turns out that the two _RE_DATASET and _RE_SNAPSHOT regex in the restrict_commands.sh above don't account for capital letters in either the dataset or snapshot, and syncoid was trying to send snapshots that started with TEMP_.

Changing those two lines to:

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'

resolved my issue. I'm now happily pulling snapshots from my primary server to my backup server. @Mikesco3 harder to tell if this resolves your issue specifically, but @alembiq I do see a capital GMT in your log output so you might give it a shot.

me bad, i never used the script above, I got this error from using syncoid from default package, but I'm getting the error above

should have read the thread more in detail before posting my question :( sorry

I had to do the same as @lediur but I also had to add two entries to the whitelist because my command wasn't matching the 2>/dev/null part of the whitelist. My final file that worked for me was:

#!/bin/bash
export PATH=/usr/sbin:$PATH

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'

_WHITELIST=(
   -e "exit"
   -e "echo -n"
   -e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
   -e "zfs get -H syncoid:sync $_RE_DATASET"
   -e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
   -e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE)?"
   -e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE)?"
   -e "command -v (lzop|mbuffer)"
)

## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
    | ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
    | logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash