I am trying to merge a branch in Reproducible Data Science | Open Research | Renku and got a warning that the last build failed. When trying to re-build, an error pops up related to updating renku:
…
Step 6/6 : RUN if [ -n “$RENKU_VERSION” ] ; then source .renku/venv/bin/activate ; currentversion=$(renku --version) ; if [ “$RENKU_VERSION” != “$currentversion” ] ; then pip uninstall renku ; gitversion=$(echo “$RENKU_VERSION” | sed -n "s/+.[[:digit:]]+.[[:digit:]]+(.dev[[:digit:]]+)(+g([a-f0-9]+))(+dirty)$/\3/p") ; if [ -n “$gitversion” ] ; then pip install --force “git+https://github.com/SwissDataScienceCenter/renku-python.git@$gitversion” ; else pip install --force renku==${RENKU_VERSION} pyshacl==0.17.0.post1 ; fi fi fi
—> Running in 51065326a089
Found existing installation: renku 0.16.0
Uninstalling renku-0.16.0:
Would remove:
/home/jovyan/.renku/venv/bin/renku
/home/jovyan/.renku/venv/lib/python3.9/site-packages/renku-0.16.0.dist-info/
/home/jovyan/.renku/venv/lib/python3.9/site-packages/renku/*
/home/jovyan/.renku/venv/lib/python3.9/site-packages/tests/*
Proceed (Y/n)? ERROR: Exception:
Traceback (most recent call last):
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py”, line 173, in _main
status = self.run(options, args)
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/commands/uninstall.py”, line 93, in run
uninstall_pathset = req.uninstall(
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/req/req_install.py”, line 633, in uninstall
uninstalled_pathset.remove(auto_confirm, verbose)
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/req/req_uninstall.py”, line 378, in remove
if auto_confirm or self._allowed_to_proceed(verbose):
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/req/req_uninstall.py”, line 419, in _allowed_to_proceed
return ask('Proceed (Y/n)? ', (‘y’, ‘n’, ‘’)) != ‘n’
File “/home/jovyan/.renku/venv/lib/python3.9/site-packages/pip/_internal/utils/misc.py”, line 203, in ask
response = input(message)
EOFError: EOF when reading a line
Collecting renku==0.16.2
Downloading renku-0.16.2-py2.py3-none-any.whl (2.0 MB)
…
Should I try to upgrade renku first?
Thanks for your help!
Cheers
Stan
It looks like it’s due to pip asking for user confirmation when uninstalling renku in that step, pip uninstall renku should probably read pip uninstall -y renku.
Thanks, @ralf.grubenmann! But is there a way I can get around this, or do I have to wait for the next update? I assume that others should have encountered this problem, too.
oh I meant you can change that command in the Dockerfile in your project and it should work.
It’s a bit weird in that I see
Step 6/6 : RUN [...] then pip uninstall renku ; [...]
in the log you pasted above, but looking on the Dockerfile in the repo the line properly contains the -y.
Also the renku version in your Dockerfile is 1.0.6 but the message above tries to install 0.16.2. Does the branch you want to merge have an older version of the Dockerfile?
Thanks! Sorry, for some reason I pasted the wrong repo link. I was trying to merge Stan_revision1 into master in the project https://renkulab.io/projects/remko.nijzink/budyko_paper. The Dockerfile on Stan_revision1 states: ARG RENKU_VERSION=0.16.2, same as in master. Should I update the project before merging? Automatic update failed, so I have to figure out how to do a manual update.
For a successful merge, both branches should be on the same version of renku (or at least the same minor version). So you’d have to update both branches before merging.
But I think you can just add the -y to the pip uninstall renku command in the Dockerfile and the build should pass.
One small caveat, renku also updates your project if the project template it was created from is changed. And there it takes care not to overwrite changes you might have done. So manually modifying the Dockerfile would mean that future updates wouldn’t update the Dockerfile with changes from the template. You can just live with that, or, once you fixed the issue and updated the project, you could force apply the current template to bring it back to a clean state using renku template set --force --interactive , but this is only available in renku >= 1.2.0.
For updating, you can try to do renku migrate in terminal and check if that gives any meaningful errors, so we can see why updating doesn’t work.
Hm, unfortunately, renku migrate did not work:
base 
~ work ❯ budyko_paper master $ renku migrate
Error: Project is not supported by this version of Renku.
Upgrade to the latest version of Renku.
This is strange, as I ran this command on renkulab, which has the latest version of Renku, right?
it’s whichever version is installed in the session, you’d still have to do pip install --upgrade renku to get a new version in the session
Aaah, thank you! Now we are potentially getting more useful errors when executing renku migrate:
base ~ work ❯ budyko_paper master 
$ renku migrate
Applying migration m_0010__metadata_fixes…
Migrating old namespaces
Fixing activity ids
Error: Couldn’t execute migration
Traceback (most recent call last):
File “/opt/conda/lib/python3.9/site-packages/renku/core/migration/migrate.py”, line 173, in migrate_project
module.migrate(migration_context)
File “/opt/conda/lib/python3.9/site-packages/renku/core/migration/m_0010__metadata_fixes.py”, line 60, in migrate
migrate_remote_entity_ids()
File “/opt/conda/lib/python3.9/site-packages/renku/core/migration/m_0010__metadata_fixes.py”, line 197, in migrate_remote_entity_ids
fix_dataset_files_based_on(dataset)
File “/opt/conda/lib/python3.9/site-packages/renku/core/migration/m_0010__metadata_fixes.py”, line 183, in fix_dataset_files_based_on
file.based_on.id = file.based_on.id.replace(“/remote-entity//”, “/remote-entities/”)
AttributeError: id
That does look like there’s something wrong with your metadata. Can you give me access to the project so I can take a closer look?
Thanks for looking into this! I sent you an invitation. For some reason, I am not able to change the visibility of the project to anything else than private. Do you know what might be the reason? Also when trying to fork it into the wave namespace, I get a renku API error.
Sorry for the delay, I couldn’t look into this earlier for private reasons.
It turns out that there was a rare edge case in the migration that the project ran into, I’ve fixed it in our code.
With that fix I’ve manage to migrate successfully. If you want, I could push that to a separate branch on your repo when the current maintenance on renkulab is finished.
Hi Ralf, Thanks so much! Yes, if you could push it to a new branch, I could then merge. No rush, hope everything is good.
I’ve pushed it to the fixed-with-migration branch (based on the previous master). I also updated the project from the template so it has a newer version of the Dockerfile.
Though that just makes sure you master is up to date and new branches created from it should be able to be merged back. But for the branch you posted about originally, this wouldn’t help, if anything, merging my branch to master will make merging your branch even harder.
I could easily do the same steps on your branch (if I knew the name) and you can merge that one instead of the one I made, solving the original issue you had 
Thanks, @ralf.grubenmann! Sorry to make you do this, but if you could, this would be great. The branch I was trying to merge was Stan_revision1, which is in the same repo. I will not merge your branch, then?
Absolutely no problem, it only takes a minute to do 
I just pushed the fix on the Stan_revision1 branch and you should be able to merge it to master, then both are fixed. You can disregard/delete the fixed-with-migration branch.
If there’s conflicts just use the version from the Stan_revision1 branch
1 Like