[Discussion] What should the CI be & do?
Over the last couple of months, we (@mupuf, @dh, @vigneshraman, @gallo, @sergi, and me @eric) have been meeting regularly to work on the CI in concert. The primary purpose of our meetings is to make sure we all agree on what we should do with the CI, but the CI is not the end product here, it is just a tool we provide for you, developers, so we need your input and guidance.
We will continue to meet every two weeks, and every mesa dev is welcome to attend, as well as other interested parties (eg. distro packagers); to avoid random people joining the link and meeting time is not public, but ask any of us and we'll be happy to provide you with all the details
We have worked on defining the goals of the CI, as a guide to our work, and in particular prioritizing the work that has a big impact on these over work that affect other things that, while important, may not be as important in the big scheme of things.
Below is the list of primary needs we have identified for the direct users (mesa developers, who use the CI through MRs, forks, etc.) and indirect users (mesa users, who don't care about the existence of the CI but are affected by its results anyway).
We tried to focus on the first principles, so there are many important things that are not mentioned here because they derive from these.
User needs
User | Needs | Responsibility for fulfilling it |
---|---|---|
Mesa developers | Verify changes (functionality works & no regression) | Developers (pre-submission testing) + CI Team (robust CI) |
Mesa developers | Easy to use (triggering, test plans, results) | CI Team (Intuitive UI, helper tools, clear documentation) |
Mesa developers | Fast pipeline execution (within defined SLAs) | Farm maintainers (System optimization, monitoring) |
Mesa users | No functionality regressions | CI Team (Catch regressions) + Developers (Collaborate on triage) |
Mesa users | No significant performance regressions | CI Team (Benchmarks) + Developers (Publish probable perf impact) |
What we expect from the devs
- Pre-push checks: Developers should run pre-push checks locally, including linting, unit tests, and any applicable static analysis tools, to catch and resolve issues before pushing code to the repository.
- Self-Review and Peer Review: Before submitting code for CI, developers should self-review their changes and conduct peer reviews to ensure the code adheres to project standards and guidelines.
- Selective CI runs: To conserve CI resources, use CI directives to run only the necessary tests based on the changes made, especially when iterating on fixes.
- Understanding CI limits: Be aware of the project's CI capacity and usual usage throughout the day, and avoid unnecessary runs that could strain resources.
- Local-first debugging: Upon a CI failure, developers should first attempt to reproduce the issue locally and perform initial debugging efforts to pinpoint the cause, using CI logs and artifacts to guide their investigation, before re-running the CI once they believe they have fixed the issue.
How do we know if we've met the requirements?
- Let's define more monitoring metrics aside from the queue SLAs.
- False positive rates: pipeline failed but not due to the actual changes
- False negative rates: pipeline succeeded despite introducing a regression
- CI hotline: Tell us about your issues on OFTC's #freedesktop, or file an issue with the CI label in https://gitlab.freedesktop.org/mesa/mesa/-/issues
- We should make a periodic survey of developers:
- Are currently known issues being improved (or planned to) to your satisfaction?
- Are there other issues we should be aware of?
- What else?
Over to you, developers (and interested parties)
- Does the above look reasonable to you?
- Did we forget something important?
Please submit your ideas as comments here.
Try to only leave one idea per thread so that people can
Once the list of requirements is accepted as being relatively complete, we (both CI devs and Mesa devs) will propose ideas of how to better meet them, through separate issues that will be linked back here.