Hi all,
I just started a new project, and in the Dockerfile, these are the lines for the installation of the last Renku version.
# RENKU_VERSION determines the version of the renku CLI
# that will be used in this image. To find the latest version,
# visit https://pypi.org/project/renku/#history.
ARG RENKU_VERSION=0.16.0
########################################################
# Do not edit this section and do not add anything below
# Install renku from pypi or from github if it's a dev version
RUN if [ -n "$RENKU_VERSION" ] ; then \
currentversion=$(pipx list | sed -n "s/^\s*package\srenku\s\([^,]\+\),.*$/\1/p") ; \
if [ "$RENKU_VERSION" != "$currentversion" ] ; then \
pipx uninstall renku ; \
gitversion=$(echo "$RENKU_VERSION" | sed -n "s/^[[:digit:]]\+\.[[:digit:]]\+\.[[:digit:]]\+\(\.dev[[:digit:]]\+\)*\(+g\([a-f0-9]\+\)\)*\(+dirty\)*$/\3/p"); \
if [ -n "$gitversion" ] ; then \
pipx install --force "git+https://github.com/SwissDataScienceCenter/renku-python.git@$gitversion" ;\
else \
pipx install --force renku==${RENKU_VERSION} ;\
fi \
fi \
fi
When Dockers then tries to create the environment, the creation fails, prompting the following errors.
---> Running in aa0cf07c9a20
uninstalled renku! ✨ 🌟 ✨
ERROR: Command errored out with exit status 1:
command: /home/jovyan/.local/pipx/venvs/renku/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-85y8wt9n/rdflib-jsonld_63ecc4cf6c7a4e458c956f0518fab575/setup.py'"'"'; __file__='"'"'/tmp/pip-install-85y8wt9n/rdflib-jsonld_63ecc4cf6c7a4e458c956f0518fab575/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-3jv2yx3n
cwd: /tmp/pip-install-85y8wt9n/rdflib-jsonld_63ecc4cf6c7a4e458c956f0518fab575/
Complete output (1 lines):
error in rdflib-jsonld setup command: use_2to3 is invalid.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f5/17/45e137be0d93b70827fe5529c0400731344a978bc792193d7d9152e6dbe4/rdflib-jsonld-0.5.0.tar.gz#sha256=4f7d55326405071c7bce9acf5484643bcb984eadb84a6503053367da207105ed (from https://pypi.org/simple/rdflib-jsonld/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement rdflib-jsonld<0.6.0,>=0.5.0 (from renku) (from versions: 0.2, 0.3, 0.4.0, 0.5.0)
ERROR: No matching distribution found for rdflib-jsonld<0.6.0,>=0.5.0
Error installing renku from spec 'renku==0.16.0'.
Hence, I am pretty convinced it is not coming from any of the packages I am trying to install. Would you help me with this? 
Thank you so much!
Best regards
Luis
2 Likes
Looks like setuptools “fixed” something… [BUG] Should fail hard when installing packages that want 2to3 · Issue #2769 · pypa/setuptools · GitHub
Pinning setuptools to <58.0.0 should fix this until it can be fixed in our dependencies.
Thanks @ralf.grubenmann !! I will then pin the version of setuptools on the requirements.txt file. I hope that helps. I will let you know if it works!
Cheers
Luis
Well, I just pin the following versions of setuptools==46.1.3 and 57.5.0. And I still get the same error:
installing renku from spec 'renku==0.16.0'...
ERROR: Command errored out with exit status 1:
command: /home/jovyan/.local/pipx/venvs/renku/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hc48hdj/rdflib-jsonld_fec895bc95cc43e0a416fb637f584d49/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hc48hdj/rdflib-jsonld_fec895bc95cc43e0a416fb637f584d49/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-hg5sb812
cwd: /tmp/pip-install-0hc48hdj/rdflib-jsonld_fec895bc95cc43e0a416fb637f584d49/
Complete output (1 lines):
error in rdflib-jsonld setup command: use_2to3 is invalid.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f5/17/45e137be0d93b70827fe5529c0400731344a978bc792193d7d9152e6dbe4/rdflib-jsonld-0.5.0.tar.gz#sha256=4f7d55326405071c7bce9acf5484643bcb984eadb84a6503053367da207105ed (from https://pypi.org/simple/rdflib-jsonld/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement rdflib-jsonld<0.6.0,>=0.5.0 (from renku) (from versions: 0.2, 0.3, 0.4.0, 0.5.0)
ERROR: No matching distribution found for rdflib-jsonld<0.6.0,>=0.5.0
Error installing renku from spec 'renku==0.16.0'.
Probably I am not pinning correctly the package. Any more hints?
Cheers
Luis
Small update. Just created a new repo, change the base image to the following
RENKU_BASE_IMAGE=renku/renkulab-cuda10.0-tf1.14:0.7.4
and got the same error. Perhaps if there is any new version of an image with CUDA, that may work.
Thanks all!
Cheers
Luis
Good morning @ralf.grubenmann ,
I experience something similar with two repositories. It seems to be independent of the specified renku version as I get a very similar error for 0.14.1 and 0.16.0
Fatal error from pip prevented installation. Full pip output in file:
/home/jovyan/.local/pipx/logs/cmd_2021-09-08_06.19.49_pip_errors.log
pip seemed to fail to build package:
rdflib-jsonld<0.6.0,>=0.5.0
Some possibly relevant errors from pip install:
ERROR: Could not find a version that satisfies the requirement rdflib-jsonld<0.6.0,>=0.5.0 (from renku) (from versions: 0.2, 0.3, 0.4.0, 0.5.0)
ERROR: No matching distribution found for rdflib-jsonld<0.6.0,>=0.5.0
Error installing renku from spec 'renku==0.14.1'.
Pinning setuptools to <58.0.0 didn’t solve it.
Thanks for investigating!
Lili
The problem is that we use pipx in the images to install renku, which uses a different environment than where requirements.txt dependencies end up in. So pinning in requirements.txt has no effect on the pipx install.
Doing /home/jovyan/.local/pipx/shared/bin/pip install setuptools==57.5.0 in the Dockerfile before the entry that installs renku works. But it seems a bit hacky, I’m not sure if there’s a better way to install something in the shared environment used by pipx.
e.g. adding a RUN /home/jovyan/.local/pipx/shared/bin/pip install setuptools==57.5.0 here should work: renku-project-template/Dockerfile at master · SwissDataScienceCenter/renku-project-template · GitHub
Inserting above line to the specified Dockerfile didn’t solve it either. I get the same error again:
Fatal error from pip prevented installation. Full pip output in file:
/home/jovyan/.local/pipx/logs/cmd_2021-09-08_07.29.26_pip_errors.log
pip seemed to fail to build package:
rdflib-jsonld<0.6.0,>=0.5.0
Some possibly relevant errors from pip install:
ERROR: Could not find a version that satisfies the requirement rdflib-jsonld<0.6.0,>=0.5.0 (from renku) (from versions: 0.2, 0.3, 0.4.0, 0.5.0)
ERROR: No matching distribution found for rdflib-jsonld<0.6.0,>=0.5.0
Error installing renku from spec 'renku==0.16.0'.
Good morning @LiliGasser and @lusamino,
the PR on setuptools which broke things has been reverted and a new release of setuptools has been tagged several hours ago. However, we’re not quite sure yet why this new version isn’t used during the build that @LiliGasser has reported and things still fail. We’re still looking into this and get back to you as soon as we have more info.
@andreas the fix they did only applies if use2_to_3 is set to false, but in rdflib-jsonld it’s set to true: rdflib-jsonld/setup.py at master · RDFLib/rdflib-jsonld · GitHub So I don’t think this fix applies to us. Plus in the discussion on the issue they explicitly mention that they want builds to fail when this is set to true.
@LiliGasser @lusamino adding the command above as a separate RUN in the Dockerfile doesn’t work, for some reason.
I tested this Dockerfile locally and that worked (adding the command on line 43 as part of the overall install command):
# For finding latest versions of the base image see
# https://github.com/SwissDataScienceCenter/renkulab-docker
ARG RENKU_BASE_IMAGE=renku/renkulab-py:3.8-0.7.5
FROM ${RENKU_BASE_IMAGE}
# Uncomment and adapt if code is to be included in the image
# COPY src /code/src
# Uncomment and adapt if your R or python packages require extra linux (ubuntu) software
# e.g. the following installs apt-utils and vim; each pkg on its own line, all lines
# except for the last end with backslash '\' to continue the RUN line
#
# USER root
# RUN apt-get update && \
# apt-get install -y --no-install-recommends \
# apt-utils \
# vim
# USER ${NB_USER}
# install the python dependencies
COPY requirements.txt environment.yml /tmp/
RUN conda env update -q -f /tmp/environment.yml && \
/opt/conda/bin/pip install -r /tmp/requirements.txt && \
conda clean -y --all && \
conda env export -n "root"
# for streamlit apps
COPY jupyter_notebook_config.py ~/.jupyter/
# RENKU_VERSION determines the version of the renku CLI
# that will be used in this image. To find the latest version,
# visit https://pypi.org/project/renku/#history.
ARG RENKU_VERSION=0.16.0
########################################################
# Do not edit this section and do not add anything below
# Install renku from pypi or from github if it's a dev version
RUN if [ -n "$RENKU_VERSION" ] ; then \
currentversion=$(pipx list | sed -n "s/^\s*package\srenku\s\([^,]\+\),.*$/\1/p") ; \
if [ "$RENKU_VERSION" != "$currentversion" ] ; then \
pipx uninstall renku ; \
/home/jovyan/.local/pipx/shared/bin/pip install setuptools==57.5.0; \
gitversion=$(echo "$RENKU_VERSION" | sed -n "s/^[[:digit:]]\+\.[[:digit:]]\+\.[[:digit:]]\+\(\.dev[[:digit:]]\+\)*\(+g\([a-f0-9]\+\)\)*\(+dirty\)*$/\3/p"); \
if [ -n "$gitversion" ] ; then \
pipx install --force "git+https://github.com/SwissDataScienceCenter/renku-python.git@$gitversion" ;\
else \
pipx install --force renku==${RENKU_VERSION} ;\
fi \
fi \
fi
########################################################
5 Likes
Thanks @ralf.grubenmann !! It worked! The correct version of setuptools is installed, and then, renku. All is back up and running.
Thank you so much
Cheers
Glad that worked! We’re also working on a proper solution to this issue, but that might take a bit longer.
1 Like
Also worked for me, thanks @ralf.grubenmann !