odagrun Docker Registry API



Registry Push


purpose

Create or Transfer a docker image on a registry from a rootfs, an existing archive or a docker image, without docker, daemonless and unpriviledged without sudo or root.

As an integrated command of the odagrun there is no need to load a specific docker image to create, copy or tag an image, actually a build image as scratch will do.

Registry_push is capable to communicate with a any docker registry without the need for docker and supports docker API V2 schemaVersion 1 & 2 manifests, while it authorizes automatically with the integrate local registry ImageStream,the GitLab-registry, DockerHub or the command line CREDENTIALS option.

As an extra benifit, registry_push provides effortless automatic labeling of the images, with com.odagrun schema and the standard OCI image labeling scheme for easy tracebility.

Preserving resources

With registry_push integrated we can at any given time immediatly push to the registry a Directory or an Archive representing our future docker image layer immediatly and thus avoiding uploading artifacts or having a bloated build image containing on top of our required building image a docker client.

Registry_push does not save any files to the filesystem, a filesystem is traversed once, wil calculating the content sha256 sum and the blob sha256 sum while using PATCH in 4MB blocks to upload directly to the registry. In theory this could be very fast if it was not for the fact that the distribution/registry breaks the stream, by saving the whole blob to disk or memory before calculating the sha256 sum and then push it to the storage backend!

Command Definition registry_push


registry_push [options] [pathname …]


Before we dive into all the posible command line options or the pathname.

Let’s start first with some basic example usage of registry_push.

Basic Examples


keep it real simple

We just compiled static an application and installed it in our project dir under ./rootfs/bin/myapp:

build:
  script:
  - registry_push
OR
build:
  script:
  - >
      registry_push
      [--GLR] \
      [--reference=latest]
      [--rootfs[=./rootfs]]
  • will create from ./rootfs directory a new image layer
  • and push that to registry as Gilab:latest
  • externaly availleble as $CI_REGISTRY_IMAGE:latest
keep it simple
build:
  script:
  - registry_push --name=test .
  • will create from . of the current directory a new image layer
  • pushed to registry as Gilab/test:latest
  • externaly availleble as $CI_REGISTRY_IMAGE/test:latest


Create a single layer image from a defined filesystem:
build:
  script:
  - >
     registry_push
     --ISR
     --rootfs=/
     --u2g
     /opt/el7/x86_64/*.rpm
     /etc/myconfig
  • will create an image layer
  • push it to the ImageStream,
  • will be containing:
    • /opt/el7/x86_64/*.rpm
    • /etc/myconfig
  • will have same group permissions (root) as owner (non-root)
  • accesible in CI as:
    • ImageStream
    • ImageStream:latest


Create single layer from existing archive
build:
  script:
  - >
     registry_push
     --archive
     --ISR
     --name=temp1
     --reference=first
  • will create a single layer image from ./layer.tar.gz
  • push it to the ImageStream
  • accesible in CI as: ImageStream/temp1:first


Append a rootfs to an existing image

For non statically build applications, we can use a different image for building and for the actually application and add or application to the application base image.

Example .GitLab-ci.yml:

build:
  script:
  - >
     registry_push
     --from_image=centos
     [--GLR]
     --name=test
     --reference=1
     --rootfs[=./rootfs]
     /opt/el7/x86_64/*.rpm
     /etc/myconfig
  • will append a new layer to or base image centos
  • from ./rootfs
  • with content /rootfs/opt/el7/x86_64/*.rpm and /etc/myconfig
  • and pushed to GitLab registry $CI_REGISTRY_IMAGE/test:1
  • while accesible in CI as: GitLab/test:1


Transfer an Image
build:
  script:
  - >
     registry_push
     --from_image=ubuntu
     --GLR
     --name=ubuntu
     --reference=1
     --config=myubuntu.yaml
  • transfer an image from a docker repository to our Gilab-Registry/project_path
  • with modified docker config as defined in ./myubuntu.yaml
  • and pushed it to GitLab registry $CI_REGISTRY_IMAGE/ubuntu:1
  • accesible in CI as:GitLab/ubuntu:1


Transfer an Image and preserve build history labels
build:
  script:
  - >
     registry_push
     --from_image=GitLab/ubuntu:1
     --image=mynamespace/ubuntu:1
     --skip-label
  • transfer an image from our GitLab registry to dockerhub
  • skip history layer, so we keep info regarding the build in the labels from the build phase.


registry-push command options:


  • --rootfs[=<path2rootfs>] push a rootfs to registry, default ./rootfs.
  • --archive[=<path2archive.tar.gz>] push an archive as layer, default ./layer.tar.gz.
  • --from_image=<repo_path>[:tag] from a qualified reponame
  • --from_ISR default:<IS_REGISTRY>/namespace/is-project_path_slug[-name][:latest]
  • --from_GLR default:<CI_REGISTRY>/<CI_PROJECT_PATH>[/name][:latest]
  • --from_name optional name only with:
    • --from_ISR
    • --from_GLR.
  • --from_reference optional reference only with:
    • --from_ISR
    • --from_GLR
  • --from_credentials for pulling an image,
  • --image=<repo_path>[:tag] push to image.
  • --ISR defaults to <IS_REGISTRY>/<namespace>/is-<project_path_slug>[-<--name>][:latest]
  • --GLR defaults to <CI_REGISTRY>/<CI_PROJECT_PATH>[/<--name>][:latest]
  • --name optional name only valid with:
    • --ISR
    • --GLR
  • --reference optional reference only valid with:
    • --ISR
    • --GLR
  • --credentials credentials to be used to push image format:
    • format: base64("<user>:<token>")
  • --config docker config file, default: ./config.yaml
  • --u2g copies the permissions of the owner to the group.
  • --skip_label will not append a dummy layer with container command history and not update the com.odagrun.from.xxx, only valid on a pure Transfer, thus:
    • --from_xxx
    • no --config option present
    • no --rootfs option present
    • no --archive option present


Command pathname:


Files and/or directories to include in the image layer. (filter)

  • optional defaults to . when --rootfs is defined
  • supports for simple variable expansion, through wordexp(),
  • If --rootfs is present, wildcards (*) are expanded relative to what is defined in --rootfs
Warning!
pathname does not support brace expansion: e.g. registry_push usr/{bin,lib} will fail!


registry_push Functions:


Authorisation:

By default, the oc-dispatcher and the oc-executor will use all the availleble tokens at hand:

  1. for dockerHub: the secret variable DOCKER_CREDENTIALS.
  2. for Quay.io: the secret variable QUAY_CREDENTIALS.
  3. for any image mathing the CI_REGISTRY, the CI_REGISTRY_USER and CI_REGISTRY_PASSWORD.
  4. for ImageStream the service account token of the oc-dispatcher.

This default behavior can be overwritten with the options:

credentials=
eyxxxxxxxxxxxx==
for pushing an image
from_credentials=
eyxxxxxxxxxxxx==
for pulling an image

check out the variables section: How to Create Credentials


Create an image layer from a Directory or File(s):

Can be acomplished in many ways:

  1. by not defining any (from_xxx) nor --archive nor --rootfs nor pathname: this will default with root at $PWD/rootfs and pathname=.
  2. by defining only pathname, e.g. . : the layer rootfs is the current Directory.
  3. with --rootfs and pathname defined
  4. with --rootfs only:
option rootfs path
--rootfs $PWD/rootfs
--rootfs=test $PWD/test
--rootfs=/test /test
--rootfs=
$CI_PROJECT_DIR
project_dir

Create an image layer from an archive.

  • --archive defaults to layer.tar.gz at current directory
  • --archive=myarchive.tar.gz

The archive is streamed as is, while content and blob SHA256SUM is calculated for manifest and/or config creation.

Warning!
--archive cannot be used in conjunction with --rootfs nor withpathname


Create an image layer from an image.

  1. create image layers --from_image=repository_name
    • dockerhub: centos:latest
    • Full Repo Name: registry.GitLab.com/mygroup/myproject[myname][:myreference]
    • ImageStream: ImageStream[/myname][:myreference]
    • GitLab_registry: GitLab[/myname][:myreference]
  2. create image layers from ImageStream or GitLab-registry
    • Define from which registry
      • --from_ISR
      • --from_GLR
    • Optional define --from_image_name=myname
    • Optional define --from_reference=myreference defaults to latest


Docker Config

--config[=docker_config.yml]

to modify or define the image configuration, we define it in a yaml or json file.

Sample docker_config.yml

Hostname: "test"
# Delete Entrypoint if present in --from_image
Entrypoint: null
WorkDir: "/"
Cmd:
  # delete all Cmd Entries if present in --from_image
  - _*: null
  # and add new entries for Cmd
  - /bin/bash
  - -c
  - echo hello
User: ""
Volumes:
  # delete volume /home if present in --from_image, and add /build
  - /home: null
  - /build
ExportPorts:
  - 8080/tcp
  # and delete 9090/udp entry if present in --from_image
  - 9090/udp: null
Labels:
  maintainer: "$GitLab_USER_EMAIL"
  title: |
          Test Image for building with --config
  description: |
             Test Image for building and testing oc-runner :
             Push and so on
  vendor: Gioxa Ltd. HongKong                
  tags: "$IMAGE_TAGS"

Parsing of the docker_config.yml:

  • support for bash variable substitution, if the StringValue value has double quotes
  • Single quotes will force a string=> no variable substitution.
  • a number or True or False without quotes will result in a json-number and json-boolean object!

On a Merge (when using --from_image=....)

Old value’s are merged,deleted or overwritten.

  • Set key/value-item: null to delete all key:values and set the key/value-item to null.

    key/value-item: null
    
  • set key with _*: null within a key/value-item to delete all old key:value and append new values on Merge

    key/value-item:
      _*: null
      key: value
    
  • set an key/value to null to delete the key:

    key/value-item:
    key: null
    
  • set key to “” will result in key: "" thus null length string and will not delete the item!!:

    key/value-item:
    key:
    


Autolabelling


when building a docker image from a registry_push automatically adds labels for tracability or documentation:

  • For images created from a rootfs or archive:
label description
com.odagrun-version the version of odagrun used to push this image
com.odagrun.build.image-digest the build image sha256 digest reference
com.odagrun.build.image build image full name, registry/repo-name”reference
com.odagrun.build.image-nick build image nickname in GitLab-CI
com.odagrun.build.image-tag build image tag or reference
com.odagrun.build.commit.author the GitLab commit author
com.odagrun.build.commit.id GitLab commit that trigered this build
com.odagrun.build.commit.message GitLab commit message
com.odagrun.build.commit.ref GitLab commit reference, e.g. master, 0.1.0 or branch name
com.odagrun.build.job-url url to the GitLab-ci job that pushed this image
com.odagrun.build.environment GitLab environment, e.g. production
com.odagrun.build.source.tree url to the GitLab source

example:

labels:
  com.odagrun-version: 0.0.33
  com.odagrun.build.image-digest: sha256:2a61f8abd6250751c4b1dd3384a2bdd8f87e0e60d11c064b8a90e2e552fee2d7
  com.odagrun.build.image: registry.hub.docker.com/library/centos:7.4.1708
  com.odagrun.build.image-nick: centos:7.4.1708
  com.odagrun.build.image-tag: 7.4.1708
  com.odagrun.build.commit.author: Danny <danny.goossen@gioxa.com>
  com.odagrun.build.commit.id: e2b011bfbecaa4a6a843cc3014d54ed891344e37
  com.odagrun.build.commit.message: prepend project dir for workspaces if not start with /
  com.odagrun.build.commit.ref: 52-change-registry-push-content-to-allow-bash-like-expansion
  com.odagrun.build.job-url: https://GitLab.gioxa.com/deployctl/oc-runner/-/jobs/49769
  com.odagrun.build.environment: production
  com.odagrun.build.source.tree: https://GitLab.gioxa.com/deployctl/oc-runner/tree/e2b011bf
  • Additional for a registry_push with a --from_xxx, or thus from an existing image, additional labels are added:
label description
com.odagrun.build.from-image.tag From Image TAG
com.odagrun.build.from-image.digest From image manifest Digest
com.odagrun.build.from-image.nick-name From image Nick Name as used in GitLab-CI
com.odagrun.build.from-image.docker-repo From image repo name

example:

labels:
  com.odagrun.build.from-image.tag: latest
  com.odagrun.build.from-image.digest: sha256:2a61f8abd6250751c4b1dd3384a2bdd8f87e0e60d11c064b8a90e2e552fee2d7
  com.odagrun.build.from-image.nick-name: centos
  com.odagrun.build.from-image.docker-repo: registry.hub.docker.com/library/centos


Warning!
When --skip_label is defined, the com.odagrun.build.from-xxxx labels are skipped!



registry_tag_image: Remote Tag an Image


purpose:

remote tag an existing image in a registry, without downloading the image layers!

usage

  • tagging an image in a registry:

    build:
    script:
    - >
       registry_tag_image 
       --image=repo-name
       [--reference=latest]
    
  • tagging an image in our GitLab-registry:

    build:
    script:
    - >
       registry_tag_image
       --GLR
       [--name=name]  
       [--reference=latest]
    
  • tagging an image in the ImageStream:

    build:
    script:
    - >
        registry_tag_image
        --GLR
        [--name=name]  
        [--reference=latest]