enable integration testing in CI
integration tests are being added in !330 (merged), however there are issues with trying to run it in the gitlab CI. current progress towards this effort is below for reference:
The network config (including open/used ports) for the host infra container that runs the gitlab-runner can "leak" into the container used by gitlab-runner for running the integration testing job, which causes vivian to fail to create the required bridge and services in the test container to fail because ports they need are already open/used on the host system.
podman unshare --rootless-netns
provides a convenient way around this, by running vivian with a 'clean' network config. the --rootless-netns
option requires (unsurprisingly I guess) that podman is run in rootless mode, so this requires some changes to the gitlab-runner config.toml on the gateway: (EDIT: submitted: https://gitlab.freedesktop.org/mupuf/valve-infra/-/merge_requests/347, though more tweaking may be necessary)
privileged = false
devices = ["/dev/kvm", "/dev/net/tun"]
# necessary for "ip" command
cap_add = [ "CAP_NET_ADMIN", "CAP_NET_RAW"]
However there are still problems using podman unshare, when run by gitlab-runner:
podman image exists localhost:8088/valve-infra/valve-infra-container:latest || make -j1 valve-infra-container
env \
FARM_NAME="vivian-" \
GITLAB_URL="https://gitlab.freedesktop.org" \
GITLAB_REGISTRATION_TOKEN= \
./vivian/vivian --tests-run-ansible --kernel-img=tmp/linux-b2c-v0.9.8 --ramdisk=tmp/boot2container-v0.9.8-linux_amd64.cpio.xz --gateway-disk-img=tmp/disk.img --kernel-append='b2c.volume="tmp" b2c.volume="perm" b2c.hostname=vivian b2c.container="--dns=none -v tmp:/mnt/tmp -v perm:/mnt/permanent --tls-verify=false --entrypoint=/bin/init docker://10.0.2.2:8088/valve-infra/valve-infra-container:latest" b2c.ntp_peer=auto b2c.pipefail b2c.cache_device=auto net.ifnames=0 quiet' integration-tests
make[1]: Entering directory '/builds/mupuf/valve-infra'
Trying to pull docker.io/library/registry:2...
Getting image source signatures
Copying blob sha256:44c4c74a95e4772391d5c7a2016dd41623a306d27e25b7060be844fb9f4ee0e2
Copying blob sha256:5299e6f7860564fd4df7e9a224f3d05dc26d0e855fb26ac7e9d9e156cf1422b2
Copying blob sha256:74a97d2d84d94d10a270600330cb860dbdf068ba87de665aadd18f8e3535edea
Copying blob sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49
Copying blob sha256:4c2fb79b7ce6b0abd6f8448fe016019030edc289f97327fe37b8dba13a5ef16a
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:5299e6f7860564fd4df7e9a224f3d05dc26d0e855fb26ac7e9d9e156cf1422b2": processing tar file(open /etc/ca-certificates/.wh..wh..opq: invalid argument): exit status 1
make[1]: *** [Makefile:37: local-registry] Error 125
According to this upstream issue, it seems like overlayfs is being nested, which is unsupported... https://github.com/containers/podman/issues/13496 I'm not currently sure how to work around that... maybe podman unshare should not be used, and we need to create a netns before the job (in some pre script? then tear it down after the job...), and run podman with --volume to set /var/lib/container in the container to something that isn't using overlayfs (/mnt/tmp ?) ???
Once the above issue is sorted out, vivian needs to be able to connect to the VM/infra container without a password for further provisioning & for ansible to work during the testing. One idea is to run make valve-infra-container
before starting the integrated testing, and set the development
flag in ansible. ansible would then need to be patched to enable passwordless ssh for root when the development tag is set.