ci-templates

Ready to include templates for common CI jobs.

Usage

Autopep8

By including this template in your project, its Python files will be linted with autopep8 for each Merge Request:

• Only the changed files will be checked with black.
• If they are not properly formatted according to black, the CI will fail.
• An assist-MR will be created to automatically fix the formatting.
• Upon merging this assist-MR, the changed files are checked again and the CI should pass.

To use the autopep8 CI template, follow these steps:

1. Add CI Bot as a member to your project in at least the Developer role.

2. Lastly, include the following snippet in your .gitlab-ci.yml file:

   include:
     - project: 'omegacen/ci-templates'
       ref: v5
       file: '/templates/autoformat/autopep8.yml'

The variables that can be changed are:

This CI template cannot be used together with the black CI template.

Black

By including this template in your project, its Python files will be linted with black 19.10b0 for each Merge Request:

• Only the changed files will be checked with black.
• If they are not properly formatted according to black, the CI will fail.
• An assist-MR will be created to automatically fix the formatting.
• Upon merging this assist-MR, the changed files are checked again and the CI should pass.

The default line length is 120, although this can be changed via the BLACK_LINE_LENGTH variable.

To use the black CI template, follow these steps:

1. Add CI Bot as a member to your project in at least the Developer role.

2. Lastly, include the following snippet in your .gitlab-ci.yml file:

   include:
     - project: 'omegacen/ci-templates'
       ref: v5
       file: '/templates/autoformat/black.yml'

The variables that can be changed are:

This CI template cannot be used together with the autopep8 CI template.

Conda

By including this template, the CI will automatically build one or more conda recipes in the repository and upload the built package our conda channel, https://conda.astro-wise.org.

The following jobs are added to the CI pipeline by including this template

• conda_build, which builds the package.
• conda_test, which tests the built package.
• conda_upload, which upload the package to the conda channel.

All jobs are run for MRs, tags, master, develop, and the default branch. The conda_upload job is only run for tags, and for the master, develop, release branches and the default branch. For conda_upload, the branch or job needs to be protected.

To use this template, do the following. First, include one or more conda recipes in your repository. For a repository that produces a single conda package, the file name of the recipe should be conda-recipes/meta.yaml. For more information on how to make a conda recipe, see the conda-wise documentation.

Next, include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/conda/all.yml'

The complete list of variables is given below.

If set, $CONDA_PACKAGE_VERSION has to be used as the version of the conda package.

The conda_build job exposes the following environment variables to all subsequent CI jobs:

• $CONDA_PACKAGE_NAME
• $CONDA_PACKAGE_VERSION
• $CONDA_PACKAGE_BUILD_STRING

If the conda_build job builds multiple conda packages, the above variables refer to the first conda package built.

It can take long to resolve the dependencies when creating the environment for the conda_test job, especially for packages with many dependencies. The resolve can be sped up by using the solver from mamba, either by using mamba directly by setting CONDA_BUILD_COMMAND to mambabuild or by using the experimental conda feature by setting CONDA_BUILD_EXTRA_ARGS to --experimental-solver=libmamba.

Conda build

It is possible to only include the template for the build stage, in case you do not want to automatically upload your build recipe but only want to test it. In that case, you must include the conda-build.yml template instead of conda.yml:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/conda/build.yml'

Monthly Merge

This template performs an automatic release merge and tag based on a schedule (usually monthly). It performs the following steps:

• Merge the develop branch into the master branch,
• Bump the version in specified files according to CalVer YYYY.MM.MICRO,
• Create a tag on the master branch.
• Create a GitLab release associated to the tag.

The last step is done via the CI template described below, which is included in the Monthly Merge template.

To use it, follow these steps:

1. Include the following snippet in your .gitlab-ci.yml file:

   include:
     - project: 'omegacen/ci-templates'
       ref: v5
       file: '/templates/monthlymerge.yml'

2. Next, create a pipeline schedule in your project. Set it to the first of the month. This pipeline schedule should ideally be owned by CI Bot. This user should a maintainer in your repository (ask a GitLab admin to do this for you after you've created the pipeline schedule). Also add a variable called MM_RUN_JOB with a value of 1 to the pipeline schedule.

3. Next, protect the tags created by this template via a wildcard (20* will do).

4. Lastly, add the private key of CI Bot to the CI variables as a protected variable called MM_PRIVATE_KEY.

The complete list of variables is given below.

Create GitLab releases on tags

This template automatically creates a GitLab release on tags that match either a CalVer pattern or a SemVer pattern. For CalVer tags matching the pattern ^\d{4}\.\d{2}\.\d+$ you must include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/release/calver.yml'

Whereas for SemVer tags mathching the pattern ^\d+\.\d+\.\d+$, you must include the following snippet:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/release/semver.yml'

The automatically created release has the same name as the associated tag, and its release notes contain a list of merged MRs to the default branch since the previous release.

SonarQube

By including this template, SonarQube source code analysis will automatically be run on the default branch and on merge requests.

To use it, follow these steps:

1. Add CI Bot as a member to your project in at least the Developer role.

2. Lastly, include the following snippet in your .gitlab-ci.yml file:

   include:
     - project: 'omegacen/ci-templates'
       ref: v5
       file: '/templates/sonarqube.yml'

3. Optionally, you can add one or more Project badges. In your GitLab project, go to Settings -> General -> Badges. Then add the following:

   • Name: Quality Gate Status
   • Link: https://sonar.astro-wise.org/dashboard?id=%{project_path}
   • Badge image URL: https://sonar.astro-wise.org/api/project_badges/measure?project=%{project_path}&metric=alert_status

   This adds a Quality Gate badge. There are more metric available; go to the SonarQube project page and click "Get project badges" for a list.

The complete list of variables for this CI template is given below.

LaTeX

This template builds a pdf file from latex sources. For feature branches and merge requests it will also generate a pdf that highlights the differences with the default branch.

To use it, include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/latex.yml'

The filename of the main tex file is determined automatically. This can be overruled by specifying the FILENAME_TEX variable.

The complete list of variables for this CI template is given below.

Test report badge

This template generates a badge that shows the percentage of successful tests in the test suite. It requires another job earlier in the pipeline that generates a JUnit test report. The conda_test job does generates such a report, if the conda recipe includes tests.

To use this template, include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/testreport/badge.yml'

Next, add badge to the Project badges. In your GitLab project, go to Settings -> General -> Badges. Then add the following:

• Name: Test Success Rate
• Link: https://gitlab.astro-wise.org/%{project_path}/-/pipelines/%{default_branch}/latest
• Badge image URL: https://gitlab.astro-wise.org/%{project_path}/-/jobs/artifacts/%{default_branch}/raw/report.svg?job=test_report_badge

The complete list of variables for this CI template is given below.

Test report change detection

This template compares the test report of the current pipeline to the report of the previous pipeline. If there are tests that passed before but fail now, this job fails. This is particularly useful if the test suite has failing tests that are not easily fixable (like for example astro has), but you're still interested in regressions.

This template requires another job earlier in the pipeline that generates a JUnit test report. The conda_test job does generates such a report, if the conda recipe includes tests.

To use this template, include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/testreport/diff.yml'

The complete list of variables for this CI template is given below.

List changed files

This template generates list of the changed files in a Merge Request or since the last push to a branch.

To use this template, include the following snippet in your .gitlab-ci.yml file:

include:
  - project: 'omegacen/ci-templates'
    ref: v5
    file: '/templates/changedfiles/all.yml'

Jobs in later stages of the pipeline will then have access to the changed_files.log artifacts. This file contains a list of all changed files in either the MR or since that last push to the branch.

Controlling when jobs runs

By including a CI template, the jobs in it get automatically run under the right circumstances. E.g., by including the conda template, a conda package is build and tested for each MR to the main branches, and for each commit on the main branches a conda packages is build, tested, and released.

However, if you want more control over when a particular job runs, you can use the CI_AWE_RUN_<JOBNAME> and CI_AWE_SKIP_<JOBNAME> variables to run or to skip a job, respectively. The full list of variables is as follows:

In addition, you can use the CI_AWE_SKIP_ALL and CI_AWE_RUN_ALL variables to control whether any or all of these jobs run.

For example, say you want to run the conda builds also on all merge requests (not just on MRs to the main branches). You can then add the following to your ``.gitlab-ci.yml`:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_ID
      variables:
        CI_AWE_RUN_CONDA_BUILD_TEST: '1'
    - when: always

Note that the last when: always is needed in this example because otherwise no jobs would run when there is no merge request.

Or, when you only want to run the SonarQube job (and nothing else) when the target of the merge request is the default branch:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
      variables:
        CI_AWE_RUN_SONAR_MR: '1'
        CI_AWE_SKIP_ALL: '1'
    - when: always

The variable precedence is as follows (highest first):

1. CI_AWE_SKIP_<JOBNAME>
2. CI_AWE_RUN_<JOBNAME>
3. CI_AWE_SKIP_ALL
4. CI_AWE_RUN_ALL

Template versioning

The latest version of the CI templates is in the master branch. In addition there are release branches with naming convention v<X>. The master branch is always merged into the latest release branch.

Breaking changes increase the version number of the release branch. So backwards compatibility is guaranteed when you include a template from a release branch. However, including a template from the master branch might lead to breaking changes every now and then.