I am using the Gitlab-CI server to handle my CI needs, and sometimes I find workarounds that might be worth sharing.
I am maintaining several in-house gems, and I like to be able to test them fast whenever I commit something to gitlab. However, when using the ruby:2.1 docker image, much time is wasted rebuilding gem dependencies before we get to the step where we can run rake spec. I recently introduced a way to reduce the time from commit to test from 2 minutes to 5 seconds by performing some caching.
.gitlab-ci.yml
variables:
BUILD_IMAGE: image_name-$CI_BUILD_REF
stages:
- build deps
- test
- cleanup
build dependencies:
stage: build deps
script:
- touch config
- docker run -u `id -u`:`id -g` --rm -w /tmp -v ${CI_PROJECT_DIR}:/tmp -v ${CI_PROJECT_DIR}/config:/usr/local/bundle/config ruby:2.1 bundle install --deployment
- docker build -t $BUILD_IMAGE -f docker/BuildImage/Dockerfile .
cache:
paths:
- vendor/
test gem:
stage: test
script:
- docker run --rm -w /tmp/src -v ${CI_PROJECT_DIR}:/tmp/src $BUILD_IMAGE rake spec
remove build image:
stage: cleanup
script:
- docker rmi $BUILD_IMAGE
Some explanations may be necessary.
build dependencies
- touch config
Here we need to create a config file that we can later mount into the ruby:2.1 image, so that bundler is able to write to this file when we run bundle install –deployment. The reason we have to do this is that we want to run the container as the user gitlab_ci_multirunner runs as – like this:
- docker run -u `id -u`:`id -g` --rm -w /tmp -v ${CI_PROJECT_DIR}:/tmp -v ${CI_PROJECT_DIR}/config:/usr/local/bundle/config ruby:2.1 bundle install --deployment
Here we use the -u `id -u`:`id -g` to make sure that the files written to vendor is owned by the gitlab_ci_multirunner user, else the files will be owned by root, and gitlab_ci_multirunner will fail to remove them.
The resulting gems in vendor is kept available for the duration of the job, which means that we should be able to copy it into a new docker image – and best of all, gitlab_ci_multirunner will archive the vendor folder (due to the cache directive) and make it available the next time the job runs. This means that the next time we run the job, all the gems will already be available (unless we have added or updated the Gemfile/gemspec).
The next step is necessary due to the inability of gitlab_ci_multirunner of keeping the cache between build jobs:
- docker build -t $BUILD_IMAGE -f docker/BuildImage/Dockerfile .
To show what this does, we need to provide two more files for your perusal
docker/BuildImage/Dockerfile
FROM ruby:2.1
# We need to make sure that bundle looks for gems in /tmp/vendor
COPY docker/BuildImage/bundle_config /usr/local/bundle/config
# Copy the cached gems into the image
COPY vendor /tmp/vendor
# Create a directory where we can mount the source directory
RUN mkdir /tmp/src
# Allways start with bundle exec
ENTRYPOINT ["bundle","exec"]
docker/BuildImage/bundle_config
---
BUNDLE_FROZEN: '1'
BUNDLE_PATH: /tmp/vendor/bundle
BUNDLE_DISABLE_SHARED_GEMS: '1'
With these, we build an image used for this session only, where we have copied over the (preferably cached) gems, and we are ready to use it for performing the tests:
test gem:
stage: test
script:
- docker run --rm -w /tmp/src -v ${CI_PROJECT_DIR}:/tmp/src $BUILD_IMAGE rake spec
Now we just need to remember to clean up properly
remove build image:
stage: cleanup
script:
- docker rmi $BUILD_IMAGE
And there you go: after the initial build of the gems dependencies, we save in excess of 90 seconds for each run – this means that I get the results from my ci-server after approximately 15 seconds after having pushed to my feature branch.
Until gitlab_ci_multirunner gets support for sharing cached files between jobs, this is a technique that should be useable for other types of builds as well (nodejs applications come to mind).
Hopefully this will be helpful for somebody out there:)