Odagrun Artifacts Extension of GitLab-CI


Purpose

  • extend GitLab-runner functionality to allow more granulated selection of artifacts files with wildcards * and variable substitution.
  • speedup donwloading of artifacts by means of parallel downloads to increase the download speed on high latency connections.

wildcard extension for GitLab Artifacts:

The GitLab-ci yaml linter does not allow to use wildcards and/or variable substitution in the artifacts paths, to circomvent this with odagrun we use quotes and define as e.g.:

.GitLab-ci.yml build file:

build:
  stage: build
  script:
     - ....
  artifacts:
     paths:
       - '${ODAGRUN_GITVERSION}/*.tar.gz'

The above configuration will add all *.tar.gz files in directory ${ODAGRUN_GIT_VERSION} to the JOB artifacts.


Note: Always use use quotes or yaml features to fake a valid path and odagrun will glob it.


valid notations to circomvent GitLab-ci Linter:

paths:
  - >-
       ${ODAGRUN_GITVERSION}/*.tar.gz
  - '${ODAGRUN_GITVERSION}/*.tar.gz'
  - "${ODAGRUN_GITVERSION}/*.tar.gz"

parallel downloads extension.

Problem.

When setting up a private Build environment, one might be confronted with high latency between the GitLab-ci server and your local build environment causing a lot of time to be spend on downloading artifacts sequentialy.

Solution

odagrun will download the artifacts in parallel and unzip in correct order.

artifacts

odagrun has same functionality as deployctl for artifacts downloading, and uses up to 6 threads with progress notification to allow fast downloading of artifacts in parallel.


Note:
Very interesting when connection time between build-server and GitLab-server have a high latency connection.