Posts Tags Categories

Harden your Gitlab CI pipeline with DevOps Principals

    about 1163 words   6 min 
/images/jason-dent-3wPJxh-piRw-unsplash.jpg
Contents

Lately Docker introduces a rate limit for the pulls from the public repository. If you hit them, you will have to wait for 6h until you can fetch an image again.

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect. Image requests exceeding these limits will be denied until the six hour window elapses.

There might be different strategies to stop fetching images from the official Docker repository. One is to cache the fetched layers. This works quite good and will be done behind the scene. But if you are using dind in your pipeline this will not work for you. Using dind will prevent layer from leaking from one build to another.

Photo by Jason Dent on Unsplash

Using your own images from your own registry during the CI run will give you control over the image itself and adds another layer of certainty.

Keep in mind that the docker tag mechanism uses a convention and there is no strict use. Tags can be overridden and therefor might be updated. There are some image maintainer out there, who uses some tags to catch up versions. Something like this:

  • ...:latest
  • ...:3
  • ...:3.8
  • ...:3.8.76

All of those versions have the same sha256.

1
2
3
4
5

$> docker inspect bb18a5f8d736
[
    {
        "Id": "sha256:bb18a5f8d736547f9d36d69cb0a28e8b9b0cf5527c62a5298e146db13bcaf6ca",
        ...

Now if you use a version which might not be too specific, you might get updates automatically but you d not have any control over it. Maybe a faulty version was released and now your pipeline fails with you have changed anything.

So you might use the most specific version possible. But keep in mind the tags are not immutable. If the maintainer decides to update a version you will get it without notice.

To harden your CI you need to keep the image and it’s updates under your control. Only use images from your registry during the build.

To create a clone we will need to:

  1. download the image from the official docker hub
  2. upload the image to your own registry
  3. use your own images on the pipeline

All those steps should be part of a pipeline itself. So you only kick off the pipeline and then a clone is stored on you registry.

For gitlab a pipeline could be like this:

1
2
3
4
5
6
7
8
9

stages:
  - lint
  - test
  - deploy-to-registry

variables:
  REPO_NAME: $COMPANY_NAME/node
  VERSIONS: 12.16.0-stretch
  BASE_IMAGE: node

Here we have three stages in place. First a lint to check if the Dockerfile is in a good shape, then we test if an image can be created from the information inside the Dockerfile. The last step deploys the result to our registry.

The steps themselves are in a shared place so that they can be reused in other projects. We will take a look at them next. But before that we need some specific information what makes this pipeline specific for e.g. node.

REPO_NAME: will hold the information where the copied image will be stored in our registry.

VERSIONS: will hold all versions we are interested in.

BASE_IMAGE: the image from docker hub we are interested in.

First we will need to import some common information, which will be used during the steps.

1
2
3
4
5
6
7
8

variables:
  DOCKER_TLS_CERTDIR: ""
  DOCKER_DRIVER: overlay2
  DOCKER_IMAGE: /theothertim/docker
  DOCKER_IMAGE_VERSION: 19.03.6
  REPO_PORT: 10900
  NEXUS_ADDRESS: nexus.theothertim.com
  COMPANY_NAME: theothertime

Here we define some parameter for the pipeline itself. This includes driver information, TLS, Docker Client Version and where the hosted docker hub storage is. The hosted storage in my case is a nexus instance.

Next is something specific when running jobs as DinD.

1
2
3

services:
  - name: $NEXUS_ADDRESS:$REPO_PORT/$COMPANY_NAME/dind-build-container:1.0.0
    alias: docker

The linting step might be of little use, but in future we might use the image clone and add/change something to better fit for our needs.

1
2
3
4
5
6
7
8

lint:
  stage: lint
  image: $NEXUS_ADDRESS:$REPO_PORT/$COMPANY_NAME/hadolint:v1.19.0-45-gef91156-alpine
  script:
    - hadolint Dockerfile
  except:
    - master
    - /^release/.*$/

I’m using the Hadolint image to check my Docekrfile.

1
2
3
4
5
6
7
8

build_test:
  stage: test
  image: $NEXUS_ADDRESS:$REPO_PORT$DOCKER_IMAGE:$DOCKER_IMAGE_VERSION
  script:
    - for VERSION in $(echo $VERSIONS | tr ";" "\n"); do export VERSION=$VERSION && docker build --build-arg VERSION --build-arg BASE_IMAGE -t $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION . ; done
  except:
    - master
    - /^release/.*$/

The script command might look a little bit strange, so let’s break it down in multiple parts.

Here we use a for loop to iterate over all the different versions we would like to create. As a source we take the content of $VERSIONS and split it by ;. Remember in the general .gitlab-ci.yaml we put all version we would like to copy divided by a semicolon, e.g. VERSIONS=14.6;13.6. Then this information is set to an environment variable and injected into to out Dockerfile.

We run the test step only during development time and finally on the Merge/Pull-Request.

With this step all we do is checking if the versions are present on docker hub and they can be reached. If we would have some additional commands inside the Dockerfile, those would also be checked to work correctly.

And when the request is accepted the changes get merged into master.

The next step deploying the images to our local registry.

1
2
3
4
5
6
7
8

deploy:
  stage: deploy-to-nexus
  image: $NEXUS_ADDRESS:$REPO_PORT$DOCKER_IMAGE:$DOCKER_IMAGE_VERSION
  script:
    - docker login -u $DOCKER_USER -p $DOCKER_PASSWORD $NEXUS_ADDRESS:$REPO_PORT
    - for VERSION in $(echo $VERSIONS | tr ";" "\n"); do export VERSION=$VERSION && docker build --build-arg VERSION --build-arg BASE_IMAGE -t $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION . && docker push $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION ; done
  only:
    - /^release/.*$/

In this step we do the same tasks like inside the test step, but we also login into our registry, tag our images and upload those.

Here is the complete file. I normally put each of those steps in a single file and then import the file into the build pipeline. And also some common configurations. By doing this, those steps can easily be reused for other pipelines.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

include:
  - project: 'devops/docker-blueprints/commonbuildscripts'
    file: '/services.yml'
  - project: 'devops/docker-blueprints/commonbuildscripts'
    file: '/deploy-clone.yml'
  - project: 'devops/docker-blueprints/commonbuildscripts'
    file: '/test-clone.yml'
  - project: 'devops/docker-blueprints/commonbuildscripts'
    file: '/lint.yml'
  - project: 'devops/docker-blueprints/commonbuildscripts'
    file: '/variables.yml'

And here is the final version of the .gitlab-ci.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

services:
  - name: $NEXUS_ADDRESS:$REPO_PORT/$COMPANY_NAME/dind-build-container:1.0.0
    alias: docker

variables:
  DOCKER_TLS_CERTDIR: ""
  DOCKER_DRIVER: overlay2
  DOCKER_IMAGE: /theothertim/docker
  DOCKER_IMAGE_VERSION: 19.03.6
  REPO_PORT: 10900
  NEXUS_ADDRESS: nexus.theothertim.com
  COMPANY_NAME: theothertime

stages:
  - lint
  - test
  - deploy-to-registry

variables:
  REPO_NAME: $COMPANY_NAME/node
  VERSIONS: 12.16.0-stretch
  BASE_IMAGE: node

lint:
  stage: lint
  image: $NEXUS_ADDRESS:$REPO_PORT/$COMPANY_NAME/hadolint:v1.19.0-45-gef91156-alpine
  script:
    - hadolint Dockerfile
  except:
    - master
    - /^release/.*$/

build_test:
  stage: test
  image: $NEXUS_ADDRESS:$REPO_PORT$DOCKER_IMAGE:$DOCKER_IMAGE_VERSION
  script:
    - for VERSION in $(echo $VERSIONS | tr ";" "\n"); do export VERSION=$VERSION && docker build --build-arg VERSION --build-arg BASE_IMAGE -t $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION . ; done
  except:
    - master
    - /^release/.*$/

deploy:
  stage: deploy-to-nexus
  image: $NEXUS_ADDRESS:$REPO_PORT$DOCKER_IMAGE:$DOCKER_IMAGE_VERSION
  script:
    - docker login -u $DOCKER_USER -p $DOCKER_PASSWORD $NEXUS_ADDRESS:$REPO_PORT
    - for VERSION in $(echo $VERSIONS | tr ";" "\n"); do export VERSION=$VERSION && docker build --build-arg VERSION --build-arg BASE_IMAGE -t $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION . && docker push $NEXUS_ADDRESS:$REPO_PORT/$REPO_NAME:$VERSION ; done
  only:
    - /^release/.*$/

And here is the Dockerfile:

1
2
3
4
5

ARG VERSION
ARG BASE_IMAGE

FROM $BASE_IMAGE:$VERSION

Now we can use our copied images inside the build pipeline. This avoids reaching out to the official Docker Hub repo to download the build image. This will the pipeline prevent from failing, when images can not be downloaded from Docker Hub. Also the time to download the image can be reduced, when the hosted registry and the runner are close together. Maybe connected via a switch and not over the internet.

You have learned how a build pipeline can benefit from keeping a copy of the image on a self controlled hub. By using those images, the pipeline became protected against unknown changes to the images and/or unreachable resources. Also the download time for he images can be reduced if the connection is moved from an internet based one to a direct connection.

The article was updated on 2020-12-27
 Git-Lab Ci, Hardening, Docker
Back | Home