# Linux-media CI

This is a set of scripts to test and validate the media subsystem of the Linux
Kernel. It is meant to help contributors, committers and maintainers of our
beloved subsystem.

[[_TOC_]]

## Preparation

The scripts can be used in four different ways:

-   Gitlab
-   Public containers
-   Self generated container
-   Locally

### Gitlab

This is the simplest way to run your tests.

You need a Gitlab project that forks the Linux kernel, configured the following way:

- Settings > CI/CD > General pipelines > CI/CD configuration file >
   - `gitlab/media-committers.yml@linux-media/media-ci`:  if your project lives in fdo.org
   - `https://gitlab.freedesktop.org/linux-media/media-ci/-/raw/main/gitlab/media-committers.yml`:
      if your project lives outside fdo.org

- Settings > CI/CD > Runners
   - Make sure you have runners available here. `media-ci` do not have any hardware
     requirement on the runners

- Settings > CI/CD > Variables > CI/CD Variables
   - Create a variable named `FULL_CI` with value `1` if you want to run the full
     test (discouraged, and slow).
   - Create a variable named `RUN_CI_DEFAULT_BRANCH` with value `1` if you want
     to run CI also on the default branch.
   - Create a variable named `TEST_MEDIA_TARGET` if you want to test your code
     against the test-media regression test. The value of the variable will
     be passed to test-media (typically mc or all). If the variable is set to
     `bypass` the test-media regression test will not run. Please note that
     right now we are running the test without kvm, so they will take longer
     than in your machine.
   - Create the variable `V4L_UTILS_REPO` if you want to test against a custom
     `v4l-utils` repository. Eg: git://linuxtv.org/v4l-utils.git@master

Once you have your project set up, just push a new branch with your changes to
trigger a pipeline:

```
user@host:~/linux$ git remote add gitlab git@gitlab.freedesktop.org:linux-media/users/$user
user@host:~/linux$ git push gitlab HEAD:refs/heads/user/newfeature
```

Linux-media developers are encouraged to fork
`https://gitlab.freedesktop.org/linux-media/media-committers.git` under
`https://gitlab.freedesktop.org/linux-media/users/$username`. Please avoid
creating a new project without forking. It will use **much** more fdo.org
resources.

We provide a browser extension to improve the readability of the artifacts:
- [Chrome version](https://chromewebstore.google.com/detail/media-ci/adlddlnnegdakfejianplgbpcmlnhicg?hl=en&pli=1)
- [Firefox version](https://addons.mozilla.org/en-CA/firefox/addon/media-ci/)

### Public container

The repository generates a
[container](https://gitlab.freedesktop.org/linux-media/media-ci/container_registry/48202){:.external}
with all the tools required to test your kernel tree. You just need to have a
container manager on your system like podman or docker to use it. This can be
installed using your [favourite distro](https://www.debian.org/){:.external}. Eg
for Debian:

```
user@host:~/linux$ apt-get install podman
```

The container has the media-ci scripts located at `/media-ci`, and all the
dependencies are already pre-installed.

The container can be instanced like this:

```
user@host:~/linux$ podman run --rm -it -v .:/workdir registry.freedesktop.org/linux-media/media-ci/abi sh /media-ci/testsuites/abi.sh --kernel_dir /workdir --junit_dir /workdir/junit
```

Where:

-`v .:/workdir` Mounts the current directory (your kernel tree) in the folder
`/workdir` on the container

`/media-ci/testsuites/abi.sh --kernel_dir /workdir --junit_dir /workdir/junit`
is the actual test

### Self generated container

The containers can also be built locally. After cloning the `media-ci`
(this) repository to your computer run:

```
user@host:~/linux$ ~/work/media-ci$ podman build -f docker/abi/Dockerfile -t abi .
```

This will generate a container tagged `abi` on your local container
repository.

Call it like:
```
user@host:~/linux$ podman run --rm -it -v .:/workdir abi sh /media-ci/testsuites/abi.sh --kernel_dir /workdir --junit_dir /workdir/junit
```

Check the file `gitlab/media-committers.yml` for the different container names
and how they are called.

### Locally

Clone the `media-ci` repository and execute the script `third_party/setup.sh`
inside it.

The list of dependencies can be inferred by looking at
`docker/media-ci/Dockerfile`.

Call the different `test*.sh` scripts from a clean linux repository. The scripts
will run the different tests and clean-up after them if they succeed. The exit
code of the script determines if the test has passed or failed. E.g:

```
user@host:~/linux$ sh ~/media-ci/test-pahole.sh
"/home/media-ci/test-pahole.sh " Completed!
```

## Tests

The tests are grouped in `testsuites` that can be considered the top level unit
testing.

They can be found in the repository with the name `testsuites/*.sh` they call one
or more smaller tests that are named `test-*.sh`, that can be also called
directly.

When a test fails, it will have a non-zero retcode and will print the error.

When a test succeeds it will have a 0 retcode and will revert the repository to
a clean state.

Test behaviour can be modified with environment variables and arguments. Just
run the commands with the --`help` argument to get the list of valid arguments
and env variables. E.g.

```
Usage: test-pahole.sh [--help] [--kernel_dir value] [--cross_compile value] [--cross_compile32 value] [--cc value] [--arch value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory
 --cross_compile x86_64-linux-gnu-: Cross-compile prefix
 --cross_compile32 i686-linux-gnu-: Cross-compile prefix for 32-bit ABI-checks comparison
 --cc gcc: Compiler
 --arch x86_64: Kernel architecture

Environment Variables:
 --kernel_dir: KERNEL_DIR
 --cross_compile: CROSS_COMPILE
 --cross_compile32: CROSS_COMPILE32
 --cc: CC
 --arch: ARCH

```

### ABI testsuite

It validates that the ABI has not changed. It contains three tests:

-   `test-compile-all.sh`:
    -   Makes sure that all the media code is compiled.
-   `test-pahole.sh`:
    -   Compiles `pahole/pahole.c` for all the supported arches.
    -   It checks that there are not holes or padding.
    -   It checks that arm32 and x86 are identical to arm64 and x86_64
        respectively.
-   `test-abi-dumper.sh`:
    -   Autogenerates a C file that contains all the media structures.
    -   Use `abi-compliance-checker` (from `abi-dumper`) to validate that there
        is no ABI breakage.

```
Usage: testsuites/abi.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR
```

### Build testsuite

It validates that the code from the media subsystem can be built. It contains
one test:

-   `test-build.sh`:
    -   Builds the code using the provided options

The build testsuite builds for x86 and x86_64 using the allyesconfig. It also
builds using manually created config files under the `testdata/configs/` folder.

```
Usage: testsuites/build.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR
```

### LLVM testsuite

Similar to the build testsuite but using the clang compiler instead of gcc. It
builds for arm, arm64, x86, x86_64, and powerpc with the `allyesconfig` option

```
Usage: testsuites/llvm.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR

```

### Build ancient testsuite

Similar to the build testsuite but using the oldest toolchain supported by the
kernel, as per Documentation/process/changes.rst.

In this case we only build for x86_64 with the `allyesconfig` option.

```
Usage: testsuites/build-ancient.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR

```

### Bisect testsuite

It validates that a patchset can be bisectable. For every patch the following
tests are executed to evaluate its bisectability:

-   `test-build.sh`:
    -   Building allyesconfig ignoring warnings
-   `test-misc.sh`:
    -   Checks for unsafe strcpy and family

```
Usage: testsuites/bisect.sh [--help] [--kernel_dir value] [--git_origin value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory
 --git_origin origin/main: GIT ref used for reference

Environment Variables:
 --kernel_dir: KERNEL_DIR
 --git_origin: GIT_ORIGIN
```

### Checkpatch testsuite

It validates that the code passes the `checkpatch.pl` check in strict mode, as
well as some custom media rules like:

-   Device tree code is not landed via our tree.
-   The top maintainers are not directly in cc.
-   Subject starts with "media:".

```
Usage: testsuites/checkpatch.sh [--help] [--kernel_dir value] [--git_origin value] [--patches_max value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory
 --git_origin origin/main: GIT ref used for reference
 --patches_max 50: Max number of patches to be analyzed

Environment Variables:
 --kernel_dir: KERNEL_DIR
 --git_origin: GIT_ORIGIN
 --patches_max: PATCHES_MAX
```

### Doc testsuite

The documentation is formatted properly. It contains two tests:

-   `test-kernel-doc.sh`:
    -   Validates kernel-doc for all the media header files.
-   `test-spec.sh`:
    -   Creates a spec book with the media documentation.

```
Usage: testsuites/doc.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR
```

### Static testsuite

Kernel static analyzers do not introduce new warnings.

-   `test-compile-all.sh`:
    -   Checks that all the media code can be compile tested.
-   `test-sparse.sh`:
    -   Use the sparse static analyzer building x86_64 in allyesconfig.
-   `test-smatch.sh`:
    -   Use the smatch static analyzer building x86_64 in allyesconfig.
-   `test-coccinelle.sh`:
    -   Use the coccinelle static analyzer.

```
Usage: testsuites/static.sh [--help] [--kernel_dir value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory

Environment Variables:
 --kernel_dir: KERNEL_DIR
```

### Virtme testsuite

Runs the `test-media` script in a virtual machine using virtme. It runs the test
twice, in 64 bit and 32 bit compatibility mode.

```
Usage: virtme.sh [--help] [--kernel_dir value] [--junit_dir value] [--do_setup value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory
 --junit_dir /home/user/linux/junit: Output directory for junit logs and reports
 --do_setup 0: Setup the required dependencies
 --virtme_args /home/user/media-ci/third_party/v4l-utils "mc -kmemleak": Arguments for the virme script

Environment Variables:
 --kernel_dir: KERNEL_DIR
 --junit_dir: JUNIT_DIR
 --do_setup: DO_SETUP
 --virtme_args: VIRTME_ARGS
```

### Trust testsuite

Validates that patches have been properly reviewed. The list of trusted
reviewers is maintained in a separated repository. It needs to be imported using
the script `third_party/setup-committers.txt`.

The files are named: `committers.txt`, `core_committers.txt` and
`maintainers.txt`.

```
Usage: trust.sh [--help] [--kernel_dir value] [--git_origin value] [--patches_max value] [--junit_dir value] [--do_setup value]

Arguments:
 --help: show this help
 --kernel_dir /home/user/linux: Path of the kernel directory
 --git_origin origin/master: GIT ref used for reference
 --patches_max 100: Max number of patches to be analyzed
 --junit_dir /home/user/linux/junit: Output directory for junit logs and reports
 --do_setup 0: Setup the required dependencies

Environment Variables:
 --kernel_dir: KERNEL_DIR
 --git_origin: GIT_ORIGIN
 --patches_max: PATCHES_MAX
 --junit_dir: JUNIT_DIR
 --do_setup: DO_SETUP
```