question for all ci users: can we create jobs on demand (in non-merge, non-scheduled pipelines)?
The admins have requested that we do not create a pipeline with almost 500 jobs on every push, as this is creating a ton of database objects when most of them are never used.
One solution suggested would be for pipelines to contain a single job, which the ci_run_n_monitor script can execute (technically cancel & re-create) with some variables set to configure which jobs the user wants to create & run, and that job would output a new pipeline that would contain these jobs.
(This probably requires !27443 so that we can refer to the jobs we want in a consistent way)
This would further entrench the fact that the CI cannot be used via the web interface, as the jobs would not even exist anymore. There were objections in the past to making the CI need our script to be usable, and this proposal makes this worse, so before taking on the work of rewriting the entire CI, it would be good to know if these past objections have subsided (eg. people got used to using the script, or people were worried about a use-case that didn't end up being a problem, etc.).
(merge pipelines and scheduled pipelines are non-interactive so they would continue to auto-create the same jobs as we have now, and are not affected by this proposal)
Side-note, but if we do this re-write, we can also have much smarter rules for which file change should trigger which test job, using the meson dependencies directly instead of approximately rewriting them in gitlab yaml; we could also create the "container" image jobs only when the image tag was bumped, and simplify the job inheritance tree by declaring it in eg. python instead of yaml, to support whatever kind of merging we decide, instead of just "replace", and disabling farms can be simplified too and made independent of a rebase on top of the "enable/disable" commit.
Activity
This would further entrench the fact that the CI cannot be used via the web interface, as the jobs would not even exist anymore.
I tend to disagree: the idea is not to create the specific jobs on demand, but the entire pipeline on demand.
On regular users push, the pipeline consists in one single "trigger" job which needs to be manually run if needed. Once this job is run, a new child pipeline is created with the current pipeline definition. So basically we just create one more step to request gitlab in the UI to prepare the pipeline. Rest should be the same (no guarantees on
ci_run_n_monitorthough)We could also convert the whole pipeline to use child pipelines, and replace all current manual jobs with triggers, which might make things even clearer for users. Clearer because only the dependency graph will be simpler (though I don't know how much reuse we have here between different branches of the pipeline).
I tend to disagree: the idea is not to create the specific jobs on demand, but the entire pipeline on demand.
Ah, I had misunderstood what you asked, indeed!
Then that's a pretty trivial change (basically, rename
.gitlab-ci.ymlto something else, and make a new one that contains a single job that triggers a child pipeline with that renamed file, and that new job runs automatically for merge & scheduled pipelines, and is manual for the rest). I'll do that once I've finished figuring out how to restore the CI
-
I was thinking more adding a new
whenrule condition to the jobs, rule conditioned to a variable. By default, marge, scheduled don't need the new variable, but for regular devoleppor, you would need it.Then, the manual job would look like:
manually_trigger_pieline_job: variables: RUN_DOWNSTREAM_PIPELINE: 1 trigger: include: - local: .gitlab-ci.yml rule: - if: $RUN_DOWNSTREAM_PIPELINE == 1 when: never - if: <user is marge> when: never - if: <scheduled pipeline> when: never - when: manualI think this should be enough
Then that's a pretty trivial change (basically, rename .gitlab-ci.yml to something else, and make a new one that contains a single job that triggers a child pipeline with that renamed file, and that new job runs automatically for merge & scheduled pipelines, and is manual for the rest).
FWIW, the
sanityjob should still run automatically in MR pipelines (and other jobs should depend on it passing).
with some variables set to configure which jobs the user wants to create & run
This sounds like you might have slightly misunderstood @bentiss's request. The request, as I understand it, was to use a child pipeline for the actual jobs, and to require that child pipeline to be triggered manually somehow. I don't see anything about controlling which jobs will be part of the child pipeline. In other words, it's just about moving the existing jobs to a different kind of pipeline, not about changing the set of jobs.
(merge pipelines and scheduled pipelines are non-interactive so they would continue to auto-create the same jobs as we have now, and are not affected by this proposal)
True for MR pipelines triggered by Marge. MR pipelines triggered by other users (definitely those without Mesa developer role) should be treated the same as fork pipelines though, or ~half the benefit would be left on the table.
True for MR pipelines triggered by Marge. MR pipelines triggered by other users (definitely those without Mesa developer role) should be treated the same as fork pipelines though, or ~half the benefit would be left on the table.
I meant "merge pipeline" in the sense of the terminology we described at the top of https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/.gitlab-ci.yml
Basically, "merge pipeline" = pipeline that merges the MR. Pipelines in MRs before merging are "pre-merge pipelines", pipelines after merging are post-merge, and fork pipelines are something else as well.
And yes to your first point, I did indeed misunderstand the request
Edited by Eric Engestrom
-
To add a little bit of rationale, this is what we currently have on the db:
gitlab_production_ci=# SELECT relname AS "table_name", pg_size_pretty(pg_table_size(C.oid)) AS "table_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND nspname !~ '^pg_toast' AND relkind IN ('r') ORDER BY pg_table_size(C.oid) DESC LIMIT 10; table_name | table_size ----------------------------------------+------------ ci_builds_metadata | 194 GB ci_builds | 36 GB ci_build_needs | 12 GB ci_job_artifacts | 4557 MB taggings | 4211 MB ci_build_tags_100 | 3700 MB ci_pipeline_messages | 1389 MB loose_foreign_keys_deleted_records_932 | 930 MB ci_build_report_results | 884 MB ci_stages | 688 MB (10 rows)So on the 330.6 GB of
gitlab_production_ci, 194 GB is used byci_builds_metadatawhich is only the jobs definitions:gitlab_production_ci=# \d+ ci_builds_metadata Table "public.ci_builds_metadata" Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description ---------------------------+------------------------+-----------+----------+------------------------------------------------+----------+-------------+--------------+------------- project_id | integer | | not null | | plain | | | timeout | integer | | | | plain | | | timeout_source | integer | | not null | 1 | plain | | | config_options | jsonb | | | | extended | | | config_variables | jsonb | | | | extended | | | interruptible | boolean | | | | plain | | | has_exposed_artifacts | boolean | | | | plain | | | environment_auto_stop_in | character varying(255) | | | | extended | | | expanded_environment_name | character varying(255) | | | | extended | | | secrets | jsonb | | not null | '{}'::jsonb | extended | | | build_id | bigint | | not null | | plain | | | id | bigint | | not null | nextval('ci_builds_metadata_id_seq'::regclass) | plain | | | runtime_runner_features | jsonb | | not null | '{}'::jsonb | extended | | | id_tokens | jsonb | | not null | '{}'::jsonb | extended | | | partition_id | bigint | | not null | | plain | | | debug_trace_enabled | boolean | | not null | false | plain | | | exit_code | smallint | | | | plain | | | Partition of: p_ci_builds_metadata FOR VALUES IN ('100') Partition constraint: ((partition_id IS NOT NULL) AND (partition_id = '100'::bigint)) Indexes: "ci_builds_metadata_pkey" PRIMARY KEY, btree (id, partition_id) "index_ci_builds_metadata_on_build_id_and_has_exposed_artifacts" btree (build_id) WHERE has_exposed_artifacts IS TRUE "index_ci_builds_metadata_on_build_id_and_id_and_interruptible" btree (build_id) INCLUDE (id) WHERE interruptible = true "index_ci_builds_metadata_on_build_id_partition_id_unique" UNIQUE, btree (build_id, partition_id) "index_ci_builds_metadata_on_project_id" btree (project_id) Foreign-key constraints: TABLE "p_ci_builds_metadata" CONSTRAINT "fk_e20479742e_p" FOREIGN KEY (partition_id, build_id) REFERENCES p_ci_builds(partition_id, id) ON UPDATE CASCADE ON DELETE CASCADE Access method: heapSo basically, we are filling the DB with useless jobs that could be created on demand when the users need them.
Hi,
Sorry for jumping in.. I was just wondering the following:
What purpose does those have to keep CI metadata for very long term ? Isn't archiving comments from PRs and bug reports suficient? And that, can be potentially shard-ed on multiple dbs, too.
But for CI metadata, is there possible to add another column like timestamp and create a cron job that could be run on a daily basis to do DELETE from ci-* where timestimp > X months old?
Similar retention policy perhaps could also be implemented at the docker images level if there is a convention on how the tags are generated to include timestamp. Since those are not docker images used to actual deploy applications but to just contain data. Tags that are only ephemeral, I suppose that can be purged?
That would require patching gitlab, which is a lot of effort to learn enough of the internals to do it, and a lot of effort to make sure the change keeps working every time we upgrade gitlab. I don't think anyone wants to sign up for that. We're essentially limited to what gitlab provides out of the box.
