setup shadow tests in ubuntu container

setup-shadow-on-ubuntu.bash

#!/bin/bash
cat /etc/apt/sources.list
sed -i '/deb-src/d' /etc/apt/sources.list
sed -i '/^deb /p;s/ /-src /' /etc/apt/sources.list
export DEBIAN_PRIORITY=critical
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y dist-upgrade
sudo apt-get -y install ubuntu-dev-tools automake autopoint \
    xsltproc gettext eatmydata expect byacc libtool libbsd-dev \
    pkgconf tmux
su - ubuntu << EOF
git clone https://github.com/shadow-maint/shadow
cd shadow
autoreconf -v -f --install
./autogen.sh --without-selinux --disable-man --with-yescrypt
make -j4
EOF

@alejandro-colomar yeah so i personally use lxc containers, sometimes lxd containers, and vms - which is why i prefer to have a simple script that can run anywhere. So perhaps the gist should do

cd
if [ ! -d shadow ]; then
   git clone https://github.com/shadow-maint/shadow
fi

That way if I do an lxc.mount.entry = /home/serge/src/shadow home/ubuntu/shadow, or if you have docker bind mount your shadow tree, it will preserve it, else it will clone one.

Why do you do the sed(1) stuff? Have you found some broken sources.list?

Oh, I do that because I also like to do 'apt-get build-dep shadow', but I see that's not in this gist.

That's a great idea. Maybe we should rely on the Build-deps, and not call apt-get install directly (so everything needed for installing or testing shadow would be covered by apt-get build-dep shadow). And if something's missing, we should probably include it in the debian control file.

True. That would be a good approach.

It would break right now since libbsd-dev is required but Debian packaging has not yet been updated :) But that's ok...

I guess pkgconf should also be added to the Build-depends for debian's package. Without it, the check for libbsd fails.

Heh, and expect for the tests.

@alejandro-colomar yeah so i personally use lxc containers, sometimes lxd containers, and vms

I'd love to see the complete set of commands you run to set up those. I've never worked with them. :)

Although perhaps it's not as simple as docker build ... && docker run ....

• which is why i prefer to have a simple script that can run anywhere.

If we simplify it enough that it's just a bunch of lines, maybe it's not even worth a script. With the Build-deps thingy, half of the script is just a few apt-get lines, and the other half is the usual autotools stuff. Calling such a script from a Dockerfile (or GH Action file) is not going to make it much smaller, so maybe we can have duplicate scripts for each package manager and container software.

So perhaps the gist should do

cd
if [ ! -d shadow ]; then
   git clone https://github.com/shadow-maint/shadow
fi

That way if I do an lxc.mount.entry = /home/serge/src/shadow home/ubuntu/shadow, or if you have docker bind mount your shadow tree, it will preserve it, else it will clone one.

I think this would overcomplicate it, and would increase overall code.

We could maybe have several Dockerfiles, one for each package manager (maybe something like share/docker/shadow/deb.Dockerfile, share/docker/shadow/rpm.Dockerfile).

I'd prefer one per distribution, and I'm not sure about the location of the files.

Are Dockerfiles for RPM distros (Fedora, SuSE, RHEL, ...) any different at all? I expect the only difference is in the base image, which is already variable with an argument. If there's anything else different, maybe we can make it variable or maybe not. If not, sure, go for separate Dockerfiles.

About the location, I like to use FHS-like tree structures in my repos. That's why I chose ./share, which I use for the same meaning that /usr/share would have in a system. Dockerfiles are arch-independent, so that's a suitable place, I think.

I prefer the Dockerfile over github actions because I can run it locally. I'm not sure if those Dockerfiles can be reused in GH Actions or if we need to repeat everything in there.

GH Actions can use Dockerfiles directly: https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action. It increases the reusability by enabling both local and GH Action execution, so I think it's better to use Dockerfiles.

Nice.

The main point is that I should be able to test locally whatever that CI can test, to the point that CI is unnecessary. Which doesn't mean I'd remove it; it's good to have it there to simplify the workflow with one-time contributors, or to catch something if we forget to run it. But CI should be just a second safety, rather than the only one. I like being able to fully test locally.

I think the CI is more important than the local workflow, but that doesn't make any difference in this conversation as we both agree that we should enable both local and CI execution of those workflows.

:)

Without trying to convince you, I'll just say why I prefer local. It helps me compile, see the problems, and loop again until perfectly clean, very fast (much faster than going through the network, waiting to receive a notification, ...), and also I choose the interface (a terminal, much nicer than GH's or any other web GUI). After it succeeds locally, I know it's going to work in CI even before submitting (of course it may still fail, if I forgot something, or if there's some random behavior, but the chances are pretty low), so I reduce the noisy notifications that others will receive for each iteration of my tests and pushes.

--- Edit:

Oh, and another nice thing of having all tests local: you can always change the CI system; no lock-in, since it should work everywhere.

+1 for no CI system lock-in.

I'd love to see the complete set of commands you run to set up those. I've never worked with them. :)

Probably outside of the scope here mainly because I use different tools in different places :) For vm's on my main 'home' server, I use uvt-kvm, because I can do:

uvt-kvm create --memory 4096 --disk 30 rust release=jammy arch=amd64 --cpu 4 --run-script-once firstboot

For containers on my 'home' server I generally do

lxc-create -t download -n shadow -- -d ubuntu -r jammy - a amd64
lxc-start -n shadow
lxc-attach -n shadow -- << EOF
wget https://gisturl
./gist
EOF

In other environments I use lxd for both containers and vms. Those use cloud-init to specify a firstboot script.

Note that containers for shadow testsuite can be a problem as the testsuite uses a wide uid allocation.