odagrun WORK_SPACES


Purpose

a cache alternative and way to provide an unique workspace within a pipeline, providing files system and/or environment variables.

Unlike other CI-systems that have CACHE or might have WORK_SPACE, odagrun makes no distintion, it’s up to the configuration what is the WORK_SPACE function.

WORK_SPACES are actually container images and are pushed to the local registry ImageStream, which has a few distinct advantages:

  • Token-less : odagrun uses the service account, no need to define e.g. an S3 account etc…
  • Free storage, as long as the registry is free.
  • Fast, when a executer-POD is started, data is present!

This is realized by creating an on the fly executor image, containing:

  • the requested build-image layer(s)
  • optional GIT_CACHE layer(s)
  • optional WORK_SPACES layer(s)
  • the executor layer

while at the end of a job, optional and depending on the treshold, pushing the in path defined paths and the in environment defined variables to a container image, with name is-executer-image and a tag with a hash representing the name,scope,project url …etc.

Example: caching a distro RPM repository.

work spaces have different function, depending how defined

Let’s start with an example:

this example shows the use of using workspaces as a repository cache and was originaly taken from the buildimage project for this project:

.GitLab-ci.yml:

build_rootfs:
  image: gioxa/imagebuilder
  stage: build
  variables:
    GIT_CACHE_STRATEGY: push-pull
    WORK_SPACES: |
      - name: repocache centos 7.4.1708
        key: x86_64
        scope: global
        path: 
        - cache/yum/x86_64/7.4.1708/base
        - cache/yum/x86_64/7.4.1708/updates
        environment:
        - ODAGRUN_GITVERSION
        strategy: push-pull
        mandatory: false
        threshold:
          method: file_count
          path: 
          - cache/yum/x86_64/7.4.1708/base/packages/*.rpm
          - cache/yum/x86_64/7.4.1708/updates/packages/*.rpm
          operand: ne
  script:
  - export target="${CI_PROJECT_DIR}/rootfs"
  - export BASE="${CI_PROJECT_DIR}"
  - export OS_CONFIG=make_os.conf
  - mkdir -pv $target
  - ./make_os
  - >-
      registry_push
      --rootfs=$target
      --ISR
      --name=$CI_PIPELINE_ID
      --reference=1
  tags:
    - odagrun

path:

path may define what to include in the WORK_SPACE and supports wildcards, variable substitution(simple) but not bracket expansion yet, and is not mandatory, as we could opt only to use environment, see further.

In this example we only want base and updates, form our yum/cache, if we would reference to cache, other repositories or architectures could be included, not what we want for our global repository cache in this case.

If one chooses scope project, that would actually not matter, but every project would create a new WORK_SPACES image.

Relative paths for path are relative to the CI_PROJECT_DIR, path can be a string or array of strings.

Example:

WORK_SPACES:
  - name: test1
    path: mypath
  - name: test2
    path: 
    - mypath/${CACHE}
    - /my_other_absolute_path

this example would have the WORK_SPACE path of test1 resolve to:

/builds/project_path/mypath

and test2, with CACHE set to e.g. mycache, resolve to:

/builds/project_path/mypath/mycache
/my_other_absolute_path

scope:

scope can be set to:

  • global
  • group
  • project
  • pipeline

and defaults to project if not defined

This defines where this image is availleble, setting scope to global means every project can use this WORK_SPACE as well.

on top of that, an additional key can be set, which may also contain variables.

environment:

environment may consist of an

  • array of strings
environment:
 - VAR_A
 - VARX

OR

environment: ["VAR_A","VAR_B"]
  • string representing only one variable
environment: THIS_VAR
  • comma separated string of variables, spaces are removed
environment: VAR_A, VAR_B,VAR_C

defining which environment variables should be ported to the WORK_SPACE environment.

mandatory

when a job image is constructed, and mandatory is set to true, the job will fail if the WORK_SPACE cannot be found!

defaults to False except on a scope: pipeline, it defaults to True

threshold:

  • path: [“/.xxx”]
  • methode: filecount
  • operand: ne

to avoid to push a new work_space image on every build, we can use threshold, for now this has only one methode being filecount and opereand defaults to ne (not Equal) or an int value, if not equal or smaller the work_space push is cancelled.

In this example, we only push an image if the file-count of rpm’s in the yum cache has changed.

Example: Caching gems

from fpm-centos, caching of gems for ruby apps:

extract from .GitLab-ci.yml:

gem_install:
  image: ImageStream/$CI_PIPELINE_ID:install
  stage: gems
  variables:
    WORK_SPACES: |
          - name: gemcache
            scope: global
            path: ${HOME}/.gem/ruby/cache
            strategy: push-pull
            threshold:
              path: ${HOME}/.gem/ruby/cache/*.gem
  script:
    - gem install --no-ri --no-rdoc fpm
    - ../bin/fpm --version
    - export ODAGRUN_IMAGE_VERSION=$(../bin/fpm --version)
    - >-
        registry_push 
        --from_ISR
        --from_name=$CI_PIPELINE_ID
        --from_reference=base
        --rootfs=/
        --ISR
        --name=$CI_PIPELINE_ID
        --reference=fpm
        --config /builds/bin /builds/.gem
    # save IMAGE_VERSION for pushing tags
    - 'echo -e -n "${ODAGRUN_IMAGE_VERSION}" > .ODAGRUN_IMAGE_VERSION'
  tags:
    - odagrun-starter
  artifacts:
    paths:
     - .ODAGRUN_IMAGE_VERSION

Since the runner user is always non-root with a home in HOME or /builds we don’t have to set variables to redirect the cache.

Example: pure pipeline WORK_SPACE:

Sample .GitLab-ci.yml:

stages:
 - workspace
 - testing
 - cleanup

# global variables, for all jobs, default
# - no GIT
# - pull WORK_SPACE:pipeline mandatory

variables:
  GIT_STRATEGY: none
  WORK_SPACES: |
               - name: pipeline
                 scope: pipeline
                 strategy: pull
                 mandatory: True

# stage one, create workspace
create_wp:
  stage: workspace
  image: ruby
  variables:
    GIT_STRATEGY: fetch
    GIT_CACHE_STRATEGY: push-pull
    WORK_SPACES: |
      # pipeline workspace
      - name: pipeline
        scope: pipeline
        path: $HOME
        environment:
          - ODAGRUN_GITVERSION
        strategy: push
      # gem global cache
      - name: gemcache
        scope: global
        path: ${HOME}/.gem/ruby/cache
        strategy: push-pull
        threshold:
          path: ${HOME}/.gem/ruby/cache/*.gem
  script:
    - bundle install

.testing: &testing
  image: ruby
  script:
   - bundle rake specs/${CI_JOB_NAME}.spec

test1: *testing
testx: *testing
testy: *testing
testz: *testing

cleanup:
  stage: cleanup
  image: ruby
  variables:
    WORK_SPACES: |
            - name: pipeline
              strategy: clean
  script:
    # need a dummy command, GitLab does not allow empty script
    - cd ..
  when: always
  1. create a WORK_SPACE,
    • git fetch cached,
    • bundle install, which installs the gems, with cached gems.
    • push WORK-SPACE
  2. testing, start with a ready to go environment, as soon as pod comes online.
    • with environment variable: ODAGRUN_GITVERSION
    • with $HOME (/builds) containing all the project files and $HOME files.
  3. cleanup!!!