ci: Do no build as root inside docker

changed the description

quick googling seems to say it means you're basically root on the host...

That's only for the resources that you are allowed to touch in the namespace iirc. But in any case, its indeed stupid that docker defaults to privileged users and we should make a user in the images as well.

It's a one line in your docker file to setup the user. It can't default to anything else, users are specific to your docker.

Note that if we go that path, we'll need to some work to get the PTP setuid stuff to work. Normally you delegate this to the post install script of a package (rpm, deb). Right now, with root, it was set properly in the tarball, going non-root will mean that bit won't ever be set.

There's gst-ci!313 (closed) making it so the image for gst-build (for start) runs in a non-root context. We can do the rest afterwards.

Note that if we go that path, we'll need to some work to get the PTP setuid stuff to work. Normally you delegate this to the post install script of a package (rpm, deb).

Can you elaborate a bit more on what will be needed? The container still installs packages as root, then sets the user context, but also still has sudo access in it as well.

added CI label

Is this still relevant after gst-ci!272 (merged) (Replace docker-in-docker with buildah)?

Only the hing that comes to mind is that it would catch anything that accidentally requires root. we already know some ptp stuff does.

I think I misunderstood the gist of this ticket, sorry.

changed title from Do no build as root inside docker to ci: Do no build as root inside docker

moved from gst-ci#7 (moved)

I think @alatiera has been / is looking into this, also buildah and kata container support.

A brief summary after getting back to this:

It's a bit of a rabbithole, the current ci-templates setup is non-ideal as it always composes the image as root and adds the user at the end.

This causes ownership issues with resources we want to cache and copy/move afterwards as they are owned by root and we need to chown a bunch of things.

In addition the user ci-templates creates isn't in the wheel or sudo group so we have to add that ourselves anyway.

Something like this while composing, in addition to setting up sudo/wheel where required (like debian).

user=${FDO_USER:="container-user"}
sed -i -e 's/# %wheel/%wheel/' -e '0,/%wheel/{s/%wheel/# %wheel/}' /etc/sudoers
echo "%$user ALL = (ALL) NOPASSWD: ALL" > /etc/sudoers.d/99_fdo-user

There's some code to allow us to create the user ourselves and we'd switch when we needed in the image compose, however that seems to be bugged atm and I haven't figured out why yet.

https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/801093be0b88f23c8dbabe9cfce7134548f7ccdc/bootstrap/cbuild#L460

+ id ciuser
uid=10042(ciuser) gid=10042(ciuser) groups=10042(ciuser),10(wheel)
+ whoami
root
+ rm /tmp/tmp.pMzdZIiLwS
+ popd
+ buildah config --workingdir / --env HOME=/root fedora-working-container
/builds/alatiera/gstreamer_test_test123
+ rm -rf /var/lib/containers/storage/vfs/dir/8188792c061629156d2d419442e96a423221064e2287d7866c9eb8eeb8a4fa44/tmp/clone
+ for cmd in "${pkgcmd_clean[@]}"
+ buildah run --isolation chroot fedora-working-container sh -c 'dnf clean all'
42 files removed
+ '[' -z ciuser ']'
+ id ciuser
+ uid=10042
+ buildah run --isolation chroot fedora-working-container -- groupadd ciuser -g 10042
groupadd: group 'ciuser' already exists

Creating the user in the _EXEC script works, but once it exits id can't find it for some weird reason, even though useradd and groupadd query the state find and complain it exists. Note how it goes into the uid=10042 branch here cause id itself errors out with user not found. Should make a minimal reproducer for this and file an issue against ci-templates.

There was an issue with the default uid fixed in freedesktop/ci-templates!200 (merged)

Also the template hardcodes the HOME as an env var in the image itself, which is problematic in two ways:

First one is that a bunch of tooling relies on HOME to install stuff, (which is root atm due to the issue above) but also causes issues if you want to create a container from the image using a different uid/user as things will not be available. For python things we can install system wide and/or set PYTHONUSERBASE to some system place like /var/lib/python or even somewhere in /usr, but that doesn't work atm due to the root composition and file ownership issues. We'd also have to fix some $PATH and the like variables.

The second issue is that when you try to mount your own homedir, as toolbox will do for example, HOME=/home/ciuser persists and the shell (and everything reading the env var really) you will launch into the container will try to look there to load stuff. So almost everything will be trying following the wrong homedir.

mkdir: cannot create directory ‘/home/ciuser’: Permission denied
touch: cannot touch '/home/ciuser/.config/toolbox/toolbox-welcome-shown': Permission denied
bash: /home/ciuser/.bash_profile: Permission denied
⬢[alatiera@toolbox alatiera]$

I haven't found a way to override the env var of an image without recomposing it afterwards with something like from: image ENV HOME=. We could also fork ci-tempaltes to undhardcode HOME assuming PYTHONUSERBASE and everything else gets moved systemd wide during compose.

I will bounce off this task again, since I've run out of ideas and have spent like a week trying to hunt down all the causes and root issues here. But maybe will revisit in the future.