Multiple instances of a program with a big DB on small-ish drive

bigdb.md

The idea here is to use LVM copy-on-write snapshot feature to reduce disk usage needed for the multiple instances.

1. Create a 300G sparse file. This file will not be using much disk space initially, despite its apparent size.

truncate -s 300G image.img

2. Attach a loop device to the file and grab its name

loop=$(losetup --show -f image.img)

3. Create an LVM Physical Volume on the loop device, and make a Volume Group containing that volume

pvcreate "${loop}"
vgcreate vgloop "${loop}"

4. Create an LVM Thin Pool on the Volume Group. The need for it can be avoided by using a larger sparse file, but IMO this way fewer calculations are needed.

lvcreate -l100%FREE -T vgloop/tpool

5. Create a 250G volume for the original database.

lvcreate -V250G -T vgloop/tpool -n orig

6. Set up an ext4 filesystem on the volume and mount it. -o discard is supposed to give up freed blocks back to the host filesystem from the image file, but I'm not sure how well it works. For me, fstrim -v mountpoint/ did the trick.

mkfs.ext4 /dev/vgloop/orig
mount -o discard /dev/vgloop/orig orig/

7. Copy the database (or other files) to the "original" volume. If you have it all in a single file, this dd command will do it faster and without flushing kernel caches with large read/write operations. If you copy the file by other means, you may want to do a sync, too.

dd if=state.db of=orig/state.db bs=1G iflag=direct oflag=direct status=progress

8. Create as many thin snapshots of the original volume as you need, one for each copy of the program

lvcreate -s --name snap1 vgloop/orig
lvcreate -s --name snap2 vgloop/orig
...

9. Make the volumes activable, and before that disable "skip activation" on them, so they can be activated and later re-attached later with just losetup

lvchange -kn vgloop/snap1
lvchange -kn vgloop/snap2
...
lvchange -ay vgloop/snap1
lvchange -ay vgloop/snap2
...

10. Mount the copies

mount -o discard /dev/vgloop/snap1 snap1/
mount -o discard /dev/vgloop/snap2 snap2/
...

Now you can start multiple instances using snap1/, snap2/ and so on as data directories.

To remove snapshots:

umount snap1
umount snap2
...
lvremove -y vgloop/snap1
lvremove -y vgloop/snap2
...

To detach the volumes and deactivate VG, and also the loop device:

umount orig
umount snap1
umount snap2
...
vgchange -an vgloop
losetup -d "${loop}"

After that you can remove the image.img file, or re-activate the volumes:

losetup --show -f image.img
vgchange -ay vgloop
mount -o discard /dev/vgloop/orig orig/
mount -o discard /dev/vgloop/snap1 snap1/
mount -o discard /dev/vgloop/snap2 snap2/
...

You can check the actual disk usage of the image file like this:

du -h image.img

Note that it will differ from the apparent size you see in ls -lh output.