Skip to content

Reuse job attributes in Collate.from_pipeline to fix N+1 problem

Igor Ponomarev requested to merge igor.ponomarev/ci-collate:fix-n-plus-1-pipeline-init-copy-attrs into main

Currently initializing the CollateJob take 1 list HTTP call to get the list of ProjectPipelineJob and N number calls to fetch the ProjectJob objects. This is very inefficient and slow.

The issue with ProjectPipelineJob is that it is missing the methods like .trace() or .artifact() therefore it cannot be a replacement for ProjectJob.

Instead copy ProjectPipelineJob attributes to a new instance of ProjectJob which will be initialized without making HTTP request.

According to the OpenAPI schema ProjectJob and ProjectPipelineJob objects have the same schema:

https://gitlab.com/gitlab-org/gitlab/-/blob/45ee705265a161ac1f1cc3576db51f9523cc527f/doc/api/openapi/openapi_v2.yaml?page=8#L6609

This way the number of required HTTP calls will drastically go down.

Merge request reports