Hello, Since a few days I am experiencing an issue with CI/CD runners while building Dockerfile across multiple repositories. They particularly get stuck in the following line until the timeout is reached (even if it’s 10h): RUN mamba env update -q -f /tmp/environment.yml && \ /opt/conda/bin/…

Hi @rrrrrok , I am experiencing the same. the CI/CD fails because taking too long (limit is now 1h, but much less is needed normally), and it get stuck at the mamba env update. Same as for @firat Any update on this? Locally on a VM all works (conda, mamba is not installed on it).

Sorry about that, we’re looking into it. As a workaround in the meantime: if you can build the image locally, you could push that to the registry and it will be picked up when you try to launch a session.

thanks for the workaround. Could you give us pointers if there is some in renku docs (e.g., if there’s an example) for where we should be pushing the images to get them recognized? At the moment, I switched to merging pull requests without waiting build to succeed.

The information here is a little bit outdated wrt launching the sessions locally, but all the bits about the registry, the docker login and the image naming should be correct.

Hey @firat @lin - I’ve been looking into this for an hour or so; it seems there is some issue with the conda environments generally which is causing the conda env update to get stuck. I’ve tried a couple of basic things to no avail as yet - I’ll look into it a bit more tomorrow… [image]

@firat @Lin I just realized you’re building with conda - have you tried with mamba?

Oh wow, you are right the first project seems to be outdated. But second one (private project) is actually using mamba. Let me try to replace Dockerfile with a newer Renku dockerfile setup. Edit: Updating Dockerfile didn’t fix the issue.

@rrrrrok @seanrmurphy Just some more details in case: I am using mamba, and the project was building 4 days prior in few minutes, with no issues at all. The conda environment was consistent with no conflicts, as it was re-built from scratch after important packages had to be updated (such as torch).…

@Lin you said you could build the docker image locally. Can you try again without cache? I can no longer build them locally?!

CI/CD Runners - something has changed?

RenkuLab

rrrrrok 16 August 2023 16:03 11

could it be related to this: Add Packages to Renku project via Conda - pyGIMLi - #6 by champost

Topic		Replies	Views
Container build stuck conda env in a previously working project RenkuLab	1	206	11 September 2023
CondaEnvException: Unable to determine environment RenkuLab	2	135	29 November 2023
Building renku docker images locally	2	51	11 September 2024
Renku does not recognize conda/pip dependencies RenkuLab	4	379	19 July 2022
:snake: Addressing issues with Renku image builders Announcements	0	214	22 August 2023

CI/CD Runners - something has changed?

Related topics