Commit db5f88c8 authored by Peter Hutterer's avatar Peter Hutterer Committed by Benjamin Tissoires
Browse files

tools: add a ci-fairy tool for common tasks



Intended to simplify a number of tasks commonly done in the ci.
The currently only supported command is that to delete tags from the registry.
Signed-off-by: Peter Hutterer's avatarPeter Hutterer <peter.hutterer@who-t.net>
parent 395535ce
......@@ -137,6 +137,18 @@ check commits:
reports:
junit: results.xml
pytest ci-fairy:
extends: .pip_install
script:
- pip3 install pytest
- pip3 install .
- pytest --junitxml=results.xml
artifacts:
reports:
junit: results.xml
pages:
extends: .pip_install
stage: pages
......
ci-fairy - a CLI tool
=====================
``ci-fairy`` is a commandline helper tool for some commonly performed tasks,
primarily those executed by the CI. ``ci-fairy`` is optimized to be run from
a CI script where its arguments will be hardcoded and change very little.
It does not do a lot of error handling, expect to see exceptions where
things go wrong.
Installation
------------
As the ``ci-fairy`` is likely used within a CI job, installation via
``pip3`` directly from git is recommended: ::
pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates@12345deadbeef
The ``@12345deadbeef`` suffix specifies the git sha to check out.
The git sha should match the sha of the :ref:`included templates <templates_including>`.
Or where a local checkout is available: ::
pip3 install .
Alternatively, you may directly invoke the ``ci-fairy`` tool from within the
git tree without prior installation. Note that ``pip3`` will take care of
all dependencies, for local invocations you must install those manually.
Use of the GitLab CI Environment
--------------------------------
``ci-fairy`` will make use of the `predefined environment variables set by
GitLab CI
<https://docs.gitlab.com/ee/ci/variables/predefined_variables.html>`__ where
possible. This includes but is not limited to ``CI_PROJECT_PATH``,
``CI_SERVER_URL``, and ``CI_JOB_TOKEN``.
In most cases ``ci-fairy`` does not need arguments beyond what is specific
to the interaction with the project at the time.
Authentication
--------------
Authentication is handled automatically through the ``$CI_JOB_TOKEN``
environment variable. Where the ``ci-fairy`` tool is called outside a CI
job, use the ``--authfile`` argument to provide the path to a file
containing the value of a `GitLab private token
<https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html>`__ with 'api' access.
For example, if your private token has the value ``abcd1234XYZ``,
authentication can be performed like this: ::
$ echo "abcd1234XYZ" > /path/to/authfile
$ ci-fairy --authfile /path/to/authfile <....>
Where ``--authfile`` is used within a CI job, specificy the token as a
`GitLab predefined environment variable
<https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables>`__
of type "File". The "Key" is the file name given to ``--authfile``, the
value is the token value of your private token.
Deleting registry images
------------------------
.. note:: The ``CI_JOB_TOKEN`` does not have sufficient permissions to
delete images, ``--authfile`` is required for this task.
To delete an image from your container registry, use the ``delete-image``
subcommand. This subcommands provides three modes - deleting a specific
image tag, deleting all but an image tag or deleting all images. ::
ci-fairy delete-image --repository fedora/30 --all
ci-fairy delete-image --repository fedora/30 --exclude-tag 2020-03.11.0
ci-fairy delete-image --tag "2020-03-*"
During testing, use the ``--dry-run`` argument to ensure that no tags are
actually deleted.
......@@ -10,6 +10,7 @@ efficiently use the Gitlab CI on freedesktop.org.
templates
templates-api
ci-fairy
Indices and tables
......
#!/usr/bin/env python3
from setuptools import setup
setup(name='ci-fairy',
version='0.1',
description='Commandline tools for the freedesktop.org CI',
url='http://gitlab.freedesktop.org/freedesktop/ci-templates',
author='The fdo collective',
author_email='check.the@git.history',
license='MIT',
package_dir={'': 'tools'},
packages=[''],
py_modules=['ci_fairy'],
entry_points={
'console_scripts': [
'ci-fairy = ci_fairy:ci_fairy',
]
},
classifiers=[
'Development Status :: 3 - Alpha',
'License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6'
],
python_requires='>=3.6',
include_package_data=True,
install_requires=['python-gitlab', 'click', 'colored'],
)
......@@ -81,6 +81,18 @@ check commits:
reports:
junit: results.xml
pytest ci-fairy:
extends: .pip_install
script:
- pip3 install pytest
- pip3 install .
- pytest --junitxml=results.xml
artifacts:
reports:
junit: results.xml
pages:
extends: .pip_install
stage: pages
......
#!/usr/bin/env python3
import ci_fairy
if __name__ == '__main__':
ci_fairy.main()
#!/usr/bin/env python3
import click
import colored
import gitlab
from gitlab import Gitlab
import fnmatch
import logging
import os
import sys
class ColorFormatter(logging.Formatter):
def format(self, record):
COLORS = {
'DEBUG': colored.fg('orchid'),
'INFO': colored.attr('reset'),
'WARNING': colored.fg('salmon_1'),
'ERROR': colored.fg('red'),
'CRITICAL': colored.fg('red') + colored.bg('yellow'),
}
return COLORS[record.levelname] + super().format(record)
log_format = '%(levelname)s: %(message)s'
logger_handler = logging.StreamHandler()
if os.isatty(sys.stderr.fileno()):
logger_handler.setFormatter(ColorFormatter(log_format))
else:
logger_handler.setFormatter(logging.Formatter(log_format))
logger = logging.getLogger('ci-fairy')
logger.addHandler(logger_handler)
logger.setLevel(logging.ERROR)
@click.group()
@click.option('-v', '--verbose', count=True, help='increase verbosity')
@click.option('--gitlab-url', help='GitLab URL with transport protocol, e.g. http://gitlab.freedesktop.org')
@click.option('--authfile', help='Path to a file containing the gitlab auth token string')
@click.pass_context
def ci_fairy(ctx, verbose, gitlab_url, authfile):
class Wrapper(object):
pass
ctx.ensure_object(Wrapper)
verbose_levels = {
0: logging.ERROR,
1: logging.INFO,
2: logging.DEBUG,
}
logger.setLevel(verbose_levels.get(verbose, 0))
if gitlab_url is None:
gitlab_url = os.getenv('CI_SERVER_URL')
if not gitlab_url:
logger.error('Missing gitlab URL')
sys.exit(1)
if authfile is not None:
logger.debug('Using authfile')
token = open(authfile).read().strip()
gitlab = Gitlab(gitlab_url, private_token=token)
else:
token = os.getenv('CI_JOB_TOKEN')
if token:
logger.debug('Using $CI_JOB_TOKEN')
gitlab = Gitlab(gitlab_url, job_token=token)
else:
logger.debug('Connecting without authentication')
gitlab = Gitlab(gitlab_url)
ctx.obj.gitlab = gitlab
@ci_fairy.command()
@click.option('--repository', help='The registry repository to work on, e.g. fedora/latest')
@click.option('--project', help='Project name, e.g. freedesktop/ci-templates')
@click.option('--all', is_flag=True, help='delete all images')
@click.option('--exclude-tag', help='fnmatch pattern to exclude, i.e. to not delete')
@click.option('--tag', help='fnmatch pattern to delete')
@click.option('--dry-run', is_flag=True, help='Don\'t actually delete anything')
@click.pass_context
def delete_image(ctx, repository, project, all, exclude_tag, tag, dry_run):
'''
Delete images from the container registry.
One of --tag, --exclude-tag or --all is required.
Use --tag to delete a single tag only, use --exclude-tag to delete all
but the given tag. Use --all to delete all images.
Use with --repository to limit to one repository only.
The tool is designed to use the GitLab environment variables for user,
passwords, projects, etc. where possible.
Where other authentication is needed, use a --authfile containing the
string that is your gitlab private token value with 'api' access.
This tool will strip any whitespaces from that file and use the rest of
the file as token value.
'''
if not tag and not exclude_tag and not all:
logger.error('One of --tag, --exclude-tag, or --all is required.')
sys.exit(1)
if project is None:
project_id = os.getenv('CI_PROJECT_ID')
if project_id is None:
logger.error('Missing project identifier')
sys.exit(1)
p = ctx.obj.gitlab.projects.get(project_id)
else:
p = ctx.obj.gitlab.projects.list(search=project)
if len(p) != 1:
logger.error('Invalid or ambiguous project path')
sys.exit(1)
p = p[0]
repos = [r for r in p.repositories.list() if repository is None or repository == r.name]
for repo in repos:
logger.debug('Repository {}'.format(repo.name))
for t in repo.tags.list(all=True):
if all:
do_delete = True
elif fnmatch.fnmatchcase(t.name, exclude_tag or '\0'):
do_delete = False
elif fnmatch.fnmatchcase(t.name, tag or '\0'):
do_delete = True
elif exclude_tag:
do_delete = True
else:
do_delete = False
if do_delete:
logger.info('Deleting tag {}:{}'.format(repo.name, t.name))
if not dry_run:
try:
t.delete()
except gitlab.exceptions.GitlabDeleteError as e:
if e.response_code == 403:
logger.error('Insufficient permissions to delete tag {}:{}'.format(repo.name, t.name))
sys.exit(1)
else:
raise e
def main():
ci_fairy(obj={})
if __name__ == '__main__':
main()
#!/usr/bin/env python3
from click.testing import CliRunner
from unittest.mock import patch, MagicMock
import pytest
import os
import ci_fairy
GITLAB_TEST_URL = 'https://test.gitlab.url'
GITLAB_TEST_PROJECT_ID = '11'
GITLAB_TEST_PROJECT_PATH = 'project12/path34'
# A note on @patch('ci_fairy.Gitlab')
# because we use from gitlab import Gitlab, the actual instance sits in
# ci_fairy.Gitlab and we need to patch that instance.
@pytest.fixture
def gitlab_url():
return GITLAB_TEST_URL
@pytest.fixture
def gitlab_project_path():
return GITLAB_TEST_PROJECT_PATH
@pytest.fixture
def gitlab_default_env():
return {
'CI_SERVER_URL': GITLAB_TEST_URL,
'CI_PROJECT_PATH': GITLAB_TEST_PROJECT_PATH,
'CI_PROJECT_ID': GITLAB_TEST_PROJECT_ID,
}
def mock_gitlab(gitlab):
repos = []
for i in range(3):
repo = MagicMock()
repo.name = 'repository/{}'.format(i)
repos.append(repo)
tags = []
for t in range(5):
tag = MagicMock()
tag.name = 'tag-{}'.format(t)
tags.append(tag)
repo.tags = MagicMock()
repo.tags.list = MagicMock(return_value=tags)
project = MagicMock()
project.name = GITLAB_TEST_PROJECT_PATH
project.id = GITLAB_TEST_PROJECT_ID
project.repositories.list = MagicMock(return_value=repos)
ctx = gitlab(GITLAB_TEST_URL)
ctx.projects.list = MagicMock(return_value=[project])
# This always returns our project, even where the ID doesn't match. Good
# enough for tests but may cause false positives.
ctx.projects.get = MagicMock(return_value=project)
return ctx, project, repos
def test_missing_url(caplog):
args = ['delete-image'] # need one subcommand
# we can't delete them, but the empty string is good enough
env = os.environ.copy()
for key in list(env.keys()):
if key.startswith('CI_'):
env[key] = ''
runner = CliRunner(env=env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 1
assert 'Missing gitlab URL' in [r.msg for r in caplog.records]
@patch('ci_fairy.Gitlab')
def test_no_auth(gitlab, caplog, gitlab_url):
args = ['--gitlab-url', gitlab_url, 'delete-image'] # need one subcommand
# we can't delete them, but the empty string is good enough
env = os.environ.copy()
for key in list(env.keys()):
if key.startswith('CI_'):
env[key] = ''
runner = CliRunner(env=env)
runner.invoke(ci_fairy.ci_fairy, args)
gitlab.assert_called_with(gitlab_url)
@patch('ci_fairy.Gitlab')
def test_job_token_auth(gitlab, caplog, gitlab_url):
token = 'tokenval'
args = ['--gitlab-url', gitlab_url, 'delete-image'] # need one subcommand
runner = CliRunner(env={'CI_JOB_TOKEN': token})
runner.invoke(ci_fairy.ci_fairy, args)
gitlab.assert_called_with(gitlab_url, job_token=token)
@patch('ci_fairy.Gitlab')
def test_private_token_auth(gitlab, caplog, gitlab_url):
token = 'tokenval'
authfile = 'afile'
args = ['--gitlab-url', gitlab_url, '--authfile', authfile, 'delete-image'] # need one subcommand
# --authfile overrides CI_JOB_TOKEN
runner = CliRunner(env={'CI_JOB_TOKEN': 'abcd'})
with runner.isolated_filesystem():
with open(authfile, 'w') as fd:
fd.write(token)
runner.invoke(ci_fairy.ci_fairy, args)
gitlab.assert_called_with(gitlab_url, private_token=token)
@patch('ci_fairy.Gitlab')
def test_delete_image_missing_arg(gitlab, caplog, gitlab_default_env):
args = ['delete-image']
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 1
assert 'One of --tag, --exclude-tag, or --all is required.' in [r.msg for r in caplog.records]
@patch('ci_fairy.Gitlab')
def test_delete_image_invalid_project(gitlab, caplog, gitlab_default_env):
# instantiate here so ci-fairy will instantiate the same objects
ctx = gitlab(GITLAB_TEST_URL)
p1 = MagicMock()
p1.name = 'foo'
p1.id = 0
p2 = MagicMock()
p2.name = 'foo'
p2.id = 1
# empty list or more than one of projects triggers an error
for rv in [[], [p1, p2]]:
ctx.projects.list = MagicMock(return_value=rv)
args = ['delete-image', '--all', '--project', 'foo']
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 1
assert 'Invalid or ambiguous project path' in [r.msg for r in caplog.records]
@pytest.mark.parametrize('reponame', [None, 'repository-3'])
@pytest.mark.parametrize('dry_run', [True, False])
@patch('ci_fairy.Gitlab')
def test_delete_image_all(gitlab, caplog, gitlab_default_env, reponame, dry_run):
gitlab, project, repos = mock_gitlab(gitlab)
args = ['delete-image', '--all']
if reponame:
args += ['--repository', reponame]
if dry_run:
args += ['--dry-run']
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 0
for r in repos:
for t in r.tags.list():
if dry_run:
t.delete.assert_not_called()
elif reponame is None or reponame == r.name:
t.delete.assert_called()
else:
t.delete.assert_not_called()
@pytest.mark.parametrize('reponame', [None, 'repository-2'])
@patch('ci_fairy.Gitlab')
def test_delete_image_exclude_tag(gitlab, caplog, gitlab_default_env, reponame):
gitlab, project, repos = mock_gitlab(gitlab)
TAGNAME = 'tag-2'
args = ['delete-image', '--exclude-tag', TAGNAME]
if reponame:
args += ['--repository', reponame]
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 0
for r in repos:
for t in r.tags.list():
if t.name == TAGNAME or (reponame and reponame != r.name):
t.delete.assert_not_called()
else:
t.delete.assert_called()
@pytest.mark.parametrize('reponame', [None, 'repository-1'])
@patch('ci_fairy.Gitlab')
def test_delete_image_tag(gitlab, caplog, gitlab_default_env, reponame):
gitlab, project, repos = mock_gitlab(gitlab)
TAGNAME = 'tag-1'
args = ['delete-image', '--tag', TAGNAME]
if reponame:
args += ['--repository', reponame]
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 0
for r in repos:
for t in r.tags.list():
if (reponame is None or reponame == r.name) and t.name == TAGNAME:
t.delete.assert_called()
else:
t.delete.assert_not_called()
@pytest.mark.parametrize('reponame', [None, 'repository-1'])
@patch('ci_fairy.Gitlab')
def test_delete_image_tag_fnmatch(gitlab, caplog, gitlab_default_env, reponame):
gitlab, project, repos = mock_gitlab(gitlab)
TAGNAME = 'tag-*'
args = ['delete-image', '--tag', TAGNAME]
if reponame:
args += ['--repository', reponame]
runner = CliRunner(env=gitlab_default_env)
result = runner.invoke(ci_fairy.ci_fairy, args)
assert result.exit_code == 0
for r in repos:
for t in r.tags.list():
if (reponame is None or reponame == r.name):
t.delete.assert_called()
else:
t.delete.assert_not_called()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment