Hi all
I was half way through writing this up as a request when I got the idea that this should be possible even without changes to renku:
Wouldn’t it be nice to only build a new image if the changes actually require it? The situation with pinned images is already a lot better from last year where every change by everyone caused a new build. But it’s still annoying to first build an image, and then go back to pin it and disable the build again.
A bit of trickery with the CI file to only build on changes to install files, an additional “latest” tag to the resulting image, and pinning that same latest tag actually makes this possible:
.gitlab-ci.yml:
image_build:
stage: build
image: docker:stable
except:
- /^renku/autosave.*$/
only:
changes:
- Dockerfile
- install.R
- environment.yml
- requirements.txt
- .gitlab-ci.yml
before_script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN http://$CI_REGISTRY
script: |
CI_COMMIT_SHA_7=$(echo $CI_COMMIT_SHA | cut -c1-7)
docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7 --tag $CI_REGISTRY_IMAGE:latest .
docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA_7
docker push $CI_REGISTRY_IMAGE:latest
.renku/renku.ini (you can find the path in your gitlab container registry, replace the 6 letter random tag with “latest”):
image = image = registry.renkulab.io/your/project:latest
And there you ave it; automatic builds on relevant changes only and automatic uptake/pinning in new sessions.
Disclaimers: Yes I’m aware that this goes slightly against the idea of having everything explicit for reproducibility. Though it is possible to add explicit version tags as well.
Caveat 1:The renku webinterface currently doesn’t support overwriting the pinned image and I’m not sure how the git commit to image relation is maintained by renku for going back to older commits. As such you’ll have problems if you remove things from the image and try to go back to a commit that actually requires the removed stuff. (remember the latest tag from the pinned images in the renku.ini will always point to the latest image in the registry, not the one it was back then)
Caveat 1.5: In case a project gets forked the pinned image will point to the latest image in the parent repository which might change without notice in the forked projects. The flip side of that coin is that it also allows to push fixes in the image to the forked projects (e.g. the latest issue with the renku release and project paths in older base images)
Caveat 2: Of course if your project requires compilation at image build time you don’t want to use that “only: changes:” structure and with it the latest tag becomes a bit superfluous as renku already picks the latest corresponding image.
Caveat 3: I’m not exactly sure how this will interact with the autosaving feature ¯\_(ツ)_/¯
edit:I just tried it. new branches count as changed files and a new image is created, with the latest tag. I’d say that is not recommended. I added an exception clause to exclude builds on autosave branches…
Caveat 1 has implications for reproducibility (it’s slightly harder to record which image was used to produce a given result), but it can also make life a lot easier during development times with a lot of changes and when supporting things like exercise for a university course.
cheers