ZFS pull backup with minimal permissions

README.md

Usage

on system that should be backed up

• put "restrict_commands.sh" in /usr/local/bin and make it executable
• install ts, lzop and optionally mbuffer

useradd zfsbackup --create-home --system
mkdir /home/zfsbackup/.ssh
zfs allow -u zfsbackup send,hold tank/dataset
echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys
chown zfsbackup:zfsbackup /home/zfsbackup/.ssh -R

on backup server

run cronjob with:

syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@target:tank/dataset tank/dataset

restrict_commands.sh

#!/bin/bash
export PATH=/usr/sbin:$PATH

_RE_DATASET=$'[\"\']+[a-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[a-z0-9/_]+[\"\']*@[\"\']*[a-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'

_WHITELIST=(
   -e "exit"
   -e "echo -n"
   -e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
   -e "zfs get -H syncoid:sync $_RE_DATASET"
   -e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
   -e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e "command -v (lzop|mbuffer)"
)

## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
    | ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
    | logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash

Hi @csarn

It's great to hear you've got syncoid to work without enabling root ssh logins, which is what I'm trying to achieve too.

I have tried to copy your config but I've been unable to get it to work yet:

root@asteria:~# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw zfsbackup@dionysus.cs.salford.ac.uk:lxdpool/containers/hermes astarray/backups/hermes
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
CRITICAL ERROR: ssh connection echo test failed for zfsbackup@dionysus.cs.salford.ac.uk with exit code 255 at /usr/sbin/syncoid line 1427.
root@asteria:~# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshkey=/root/.ssh/id_ecdsa zfsbackup@dionysus.cs.salford.ac.uk:lxdpool/containers/hermes astarray/backups/hermes
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
zfsbackup@dionysus.cs.salford.ac.uk: Permission denied (publickey).
CRITICAL ERROR: ssh connection echo test failed for zfsbackup@dionysus.cs.salford.ac.uk with exit code 255 at /usr/sbin/syncoid line 1427.

In your example commands above, you run:

useradd zfsbackup --system

useradd doesn't create a home dir for new users by default but we need a zfsbackup home dir to store the authorized_keys file and the restrict_commands.sh script so why not run:

useradd -m zfsbackup --system

Instead to create the zfsbackup home dir at the same time?

It seems syncoid prefers to be run as root and so I created a SSH key as the root user of the destination machine and its that public key that I copied to the zfsbackup users authorized_keys on the remote machine.

I think I'm having problems because I'm running syncoid as root but the remote username is zfsbackup. I've tried specifying the path to my private key for syncoid but that hasn't worked. I have configured the .ssh directory and its files permissions correctly on both ends, I think.

Which user do you run your syncoid cron job as, if its not root? I shouldn't need to run any zfs allow commands on the destination machine if I'm running syncoid as root.

It would be great to get this process properly documented for sanoid or maybe the Arch wiki instead.

Thanks

Hi @danboid, you're right about the need to create the home directory. I added the switch to this gist.
About your use case:
I run it the same way you do, so the cronjob runs as root.
The zfsbackup user has to be created on the system that should be backed up. I clarified that in the README.md.
That also means that the "zfs allow" commands have to be issued on the machine that should be backed up.

I think your problem is the ssh connection itself, and has nothing to do with sanoid. Did you create the user on the remote system?
Also, if you add the "restrict" command script in zfsbackup's home dir, you have to make sure it is executable and on the PATH (or add the full path to the script in authorized_keys).

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location?
because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

Sorry I'm kind of new to github, but I wanted to make a few suggestions

1. On the readme.md file
   please add mkdir /home/zfsbackup/.ssh
   else the echo 'restrict,command="restrict_commands.sh" ssh-ed25519 ...' > /home/zfsbackup/.ssh/authorized_keys fails.
   

2. On the restrict_commands.sh:
   There is no mention where to place this file, I ended up creating a .local/bin folder in the zfsbackup user's home path
   mkdir -p /home/zfsbackup/.local/bin
   and adding that path in line 2 of the restrict_commands.sh so that it reads
   export PATH=$PATH:$HOME/.local/bin:/usr/sbin
   instead of export PATH=/usr/sbin:$PATH

3. Finally, if the restrict_commands.sh file fails with unknown command on line 26 error, it is because it is likely missing the ts command
   so in order to fix that I installed moreutils
   apt-get install moreutils

NOTE: just to be on the safe side it may be useful to ensure the user zfsbackup has permission to the folders we created on his home folder,
so to be safe run chown zfsbackup:zfsbackup /home/zfsbackup --recursive

Thanks for the instructions. I am able to do a transfer without the restrict_commands.sh script. However, once i enable it i get an error,

# syncoid --no-sync-snap --no-privilege-elevation --sendoptions=Rw --sshport=2222 --sshkey=/root/.ssh/mykey zfsbackup@masterserver.com:tank/stuff tank/stuff
Use of uninitialized value $sendsize in substitution (s///) at /usr/bin/syncoid line 1784.
Use of uninitialized value $sendsize in scalar chomp at /usr/bin/syncoid line 1788.
Use of uninitialized value $sendsize in pattern match (m//) at /usr/bin/syncoid line 1791.
Resuming interrupted zfs send/receive from tank/stuff to tank/stuff (~ UNKNOWN remaining):
lzop: <stdin>: not a lzop file
cannot receive: failed to read from stream
CRITICAL ERROR: ssh   -p 2222 -i /root/.ssh/mykey -S /tmp/syncoid-zfsbackup@masterserver.com-1640603651 zfsbackup@masterserver.com ' zfs send -w  -t **redacted** | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 0 |  zfs receive  -s -F 'tank/stuff' 2>&1 failed: 256 at /usr/bin/syncoid line 580.

Is the --sshport=2222 causing the shell script to fail?

if you created a user called zfsbackup shouldn't you be referencing ssh keys in the /home/zfsbackup/.ssh/mykey location? because I can't help notice that you're referencing the root's ssh key, which the zfsbackup user isn't supposed to have access to.

Well no, the ssh key usage is correct. The backup job running on the backup server has to be run as root (it is not possible on linux to allow a non-root user to "zfs receive"). So root@backup has the private key, and logs into the server-to-be-backupped using the zfsbackup user account.

Thanks for your suggestions, I updated the readme accordingly. And I'll try to find out if running ssh on a different port will cause trouble.

any luck? i'm stucked on the lzop error also :(

syncoid --sshkey=/var/lib/syncoid/kubera-2022-05-31 backup@10.10.10.10:zpool/nixos/etc tank/test --no-privilege-elevation --debug --sendoptions="w c " --sshoption=StrictHostKeyChecking=off
DEBUG: SSHCMD: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31
loginShellInit
DEBUG: checking availability of lzop on source...
DEBUG: checking availability of lzop on target...
DEBUG: checking availability of lzop on local machine...
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source zpool/nixos/etc to target tank/test.
DEBUG: getting current value of syncoid:sync on zpool/nixos/etc...
ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -H syncoid:sync ''"'"'zpool/nixos/etc'"'"''
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'tank/test' 2>&1 |"...
DEBUG: getting list of snapshots on zpool/nixos/etc using ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'zpool/nixos/etc'"'"'' |...
DEBUG: creating sync snapshot using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs snapshot ''"'"'zpool/nixos/etc'"'"''@syncoid_kubera_2024-04-01:21:54:02-GMT02:00
"...
loginShellInit
DEBUG: target tank/test does not exist.  Finding oldest available snapshot on source zpool/nixos/etc ...
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 using "ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10  zfs send -w -c  -nvP ''"'"'zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly'"'"'' 2>&1 |"...
DEBUG: sendsize = 782528
INFO: Sending oldest full snapshot zpool/nixos/etc@autosnap_2024-03-28_22:13:01_monthly (~ 764 KB) to new target filesystem:
DEBUG: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test'
DEBUG: checking to see if tank/test on  is already in zfs receive using  ps -Ao args= ...
lzop: <stdin>: not a lzop file
mbuffer: error: outputThread: error writing to <stdout> at offset 0x10000: Broken pipe
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                 ]  0%
cannot receive: failed to read from stream
mbuffer: warning: error during output to <stdout>: Broken pipe
CRITICAL ERROR: ssh  -o StrictHostKeyChecking=off  -i /var/lib/syncoid/kubera-2022-05-31 -S /tmp/syncoid-backup@10.10.10.10-1712001241 backup@10.10.10.10 ' zfs send -w -c  '"'"'zpool/nixos/etc'"'"'@'"'"'autosnap_2024-03-28_22:13:01_monthly'"'"' | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -p -t -e -r -b -s 782528 |  zfs receive  -s -F 'tank/test' failed: 256 at /usr/sbin/syncoid line 492.

I also encountered the above mysterious lzop: <stdin>: not a lzop file error. I initially tried to work around it by specifying --compress=none, but that resulted in a different cannot receive: failed to read from stream. Turns out this is a bit of a red herring.

/var/log/syslog on the remote host (source of the dataset) revealed that it was catching multiple "non-whitelisted" commands. It turns out that the two _RE_DATASET and _RE_SNAPSHOT regex in the restrict_commands.sh above don't account for capital letters in either the dataset or snapshot, and syncoid was trying to send snapshots that started with TEMP_.

Changing those two lines to:

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'

resolved my issue. I'm now happily pulling snapshots from my primary server to my backup server. @Mikesco3 harder to tell if this resolves your issue specifically, but @alembiq I do see a capital GMT in your log output so you might give it a shot.

me bad, i never used the script above, I got this error from using syncoid from default package, but I'm getting the error above

should have read the thread more in detail before posting my question :( sorry

I had to do the same as @lediur but I also had to add two entries to the whitelist because my command wasn't matching the 2>/dev/null part of the whitelist. My final file that worked for me was:

#!/bin/bash
export PATH=/usr/sbin:$PATH

_RE_DATASET=$'[\"\']+[A-Za-z0-9/_]+[\"\']+'
_RE_SNAPSHOT=$'[\"\']+[A-Za-z0-9/_]+[\"\']*@[\"\']*[A-Za-z0-9/_:-]+[\"\']+'
_RE_SIZE=$'[0-9]+[kMG]?'

_WHITELIST=(
   -e "exit"
   -e "echo -n"
   -e "zpool get -o value -H feature@extensible_dataset $_RE_DATASET"
   -e "zfs get -H syncoid:sync $_RE_DATASET"
   -e "zfs get -Hpd 1 -t snapshot guid,creation $_RE_DATASET"
   -e " *zfs send -R -w -nv?P $_RE_SNAPSHOT"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w +$_RE_SNAPSHOT \| +lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE)?"
   -e " *zfs send -R -w -nv?P +-I $_RE_SNAPSHOT $_RE_SNAPSHOT"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE 2>/dev/null)?"
   -e " *zfs send -R -w +-I $_RE_SNAPSHOT $_RE_SNAPSHOT \| lzop *(\| mbuffer +-q -s $_RE_SIZE -m $_RE_SIZE)?"
   -e "command -v (lzop|mbuffer)"
)

## LOG non-whitelisted commands, execute whitelisted
echo "$SSH_ORIGINAL_COMMAND" |\
tee >(egrep -x -v "${_WHITELIST[@]}" \
    | ts "non-whitelisted command issued: (client $SSH_CLIENT)" \
    | logger -p local0.crit \
) |\
egrep -x "${_WHITELIST[@]}" | bash