Hello, basically I would like to install the python package in my gitlab repository while building the Docker image using pip and requirements.txt
How should I add it in the requirements.txt file? Given that I do not know the relative path of my repository inside the docker image.
Thank you and best regards.
Hi Daniel,
Any package related files that you require ‘locally’ need to be inside the repository before the Dockerfile can find them. Is this local Python package already in your repository? If so, it probably has a setup.py file, in which case you can add a few commands in the Dockerfile, e.g.
COPY mypackage/ /tmp/mypackage
RUN pip install /tmp/mypackage/
which will then install the relevant package into your container.
Is this what you meant?
Gavin
Indeed that is what I meant to do and works very well.
Thank you!
Hi, sorry but after tinkering a bit I realised that it still is not what I want.
I want what would be equivalent to going inside the docker image shell and do a
pip install -e mypackage
being mypackage the python package I am developing in my gitlab repo.
so when I commit code changes to mypackage, these would be already integrated when I do a
import mypackage
If I copy in /tmp then the installation would be pointing to /tmp instead to my gitlab repository. So how could I go around doing this without needing to everytime go inside the jupyterlab shell and running pip install -e mypackage?
I want to make this as streamlined and automatized as possible for my collaborators when they start their sessions.
Thank you and best regards.
Hi Daniel,
So to understand correctly, you are hosting your Python package within a RenkuLab (Sign in to Renku) project and you want that package to be ready and up to date once your session runs?
If so, you should use a post-init.sh script which is designed to run after the container starts up. At that point you would then run pip install -e mypackage. See this example: Renku which does a similar thing.
Gavin
Hi Gavin,
I created the script, but still my package does not appear on my environment when I load the session,
I also tried having -e . in my requirements.txt file, as your example also shows.
I get the following error message:
. is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
Hi Daniel,
Could you check the logs of your session? At the top you should see information about your package getting installed (or the installation failing). You can find the logs if you click on the second icon from the top in the session frame:
Make sure to select the Jupyter Server logs.
Hi Rok,
Thank you for answering, I do not see any info in the log related to running the script or installation errors. Here I share it
2022-05-20T14:32:20.313758276Z [I 14:32:20.313 NotebookApp] Authentication of /metrics is OFF, since other authentication is disabled.
2022-05-20T14:32:20.612500687Z [W 14:32:20.612 NotebookApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
2022-05-20T14:32:22.733301636Z [W 2022-05-20 14:32:22.732 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733370650Z [W 2022-05-20 14:32:22.732 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733386467Z [W 2022-05-20 14:32:22.733 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733399383Z [W 2022-05-20 14:32:22.733 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733412017Z [W 2022-05-20 14:32:22.733 LabApp] 'cookie_secret_file' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733424563Z [W 2022-05-20 14:32:22.733 LabApp] 'allow_origin' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733436637Z [W 2022-05-20 14:32:22.733 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733448896Z [W 2022-05-20 14:32:22.733 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733496460Z [W 2022-05-20 14:32:22.733 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733510798Z [W 2022-05-20 14:32:22.733 LabApp] 'allow_remote_access' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.733523049Z [W 2022-05-20 14:32:22.733 LabApp] 'allow_remote_access' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
2022-05-20T14:32:22.743525838Z [I 2022-05-20 14:32:22.743 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
2022-05-20T14:32:22.743681556Z [I 2022-05-20 14:32:22.743 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
2022-05-20T14:32:22.893048357Z [I 14:32:22.892 NotebookApp] Serving notebooks from local directory: /home/jovyan/work/aurora
2022-05-20T14:32:22.893083415Z [I 14:32:22.892 NotebookApp] Jupyter Notebook 6.4.5 is running at:
2022-05-20T14:32:22.893987539Z [I 14:32:22.893 NotebookApp] http://daniel-2et-aurora-f8ca5ea6-0:8888/sessions/daniel-2et-aurora-f8ca5ea6/
2022-05-20T14:32:22.894082636Z [I 14:32:22.893 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2022-05-20T14:32:48.294982605Z [I 14:32:48.294 NotebookApp] 302 GET /sessions/daniel-2et-aurora-f8ca5ea6 (127.0.0.1) 0.800000ms
2022-05-20T14:32:50.905838328Z [I 14:32:50.905 NotebookApp] 301 GET /sessions/daniel-2et-aurora-f8ca5ea6/static/favicons/favicon.ico (127.0.0.1) 8.040000ms
2022-05-20T14:32:55.281209840Z [I 2022-05-20 14:32:55.280 LabApp] Build is up to date
2022-05-20T14:32:59.827900186Z [I 14:32:59.827 NotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
2022-05-20T14:32:59.836172290Z [W 14:32:59.835 NotebookApp] Notebook notebooks/Krakow_visualization.ipynb is not trusted
2022-05-20T14:33:00.292533207Z Exception in callback <TaskWakeupMethWrapper object at 0x7f70dc84a4c0>(<Future finis...ttps\r\n\r\n'>)
2022-05-20T14:33:00.292644283Z handle: <Handle <TaskWakeupMethWrapper object at 0x7f70dc84a4c0>(<Future finis...ttps\r\n\r\n'>)>
2022-05-20T14:33:00.292774265Z Traceback (most recent call last):
2022-05-20T14:33:00.292792054Z File "/opt/conda/lib/python3.9/asyncio/events.py", line 80, in _run
2022-05-20T14:33:00.292806047Z self._context.run(self._callback, *self._args)
2022-05-20T14:33:00.292820530Z RuntimeError: Cannot enter into task <Task pending name='Task-226' coro=<HTTP1ServerConnection._server_request_loop() running at /opt/conda/lib/python3.9/site-packages/tornado/http1connection.py:823> wait_for=<Future finished result=b'GET /sessio...https\r\n\r\n'> cb=[IOLoop.add_future.<locals>.<lambda>() at /opt/conda/lib/python3.9/site-packages/tornado/ioloop.py:688]> while another task <Task pending name='Task-237' coro=<MappingKernelManager.start_kernel() running at /opt/conda/lib/python3.9/site-packages/notebook/services/kernels/kernelmanager.py:176> cb=[IOLoop.add_future.<locals>.<lambda>() at /opt/conda/lib/python3.9/site-packages/tornado/ioloop.py:688]> is being executed.
2022-05-20T14:33:00.295952260Z [I 14:33:00.295 NotebookApp] Kernel started: fe84f818-bd47-49a4-afdf-1fdbad68387d, name: python3
2022-05-20T14:33:01.394908054Z [I 14:33:01.394 NotebookApp] 301 GET /sessions/daniel-2et-aurora-f8ca5ea6/static/favicons/favicon-busy-1.ico (127.0.0.1) 1.310000ms
2022-05-20T14:33:04.336051046Z [IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
2022-05-20T14:33:04.339872764Z [IPKernelApp] WARNING | No such comm: 5206b72b-585e-4ed1-baf3-a12e7eebf49a
apologies for letting this thread go silent @danieltba - it looks like there might be an issue with the post-install script running in the sessions, I will investigate.
Hi Daniel,
so you can see the setup that works in this project: Reproducible Data Science | Open Research | Renku
With this file, I see this in the logs:
2022-06-02T12:37:44.790124497Z ***************************
2022-06-02T12:37:44.790188577Z RUNNING POST INSTALL SCRIPT
2022-06-02T12:37:44.790202813Z ***************************
2022-06-02T12:37:48.287554812Z Collecting numpy
2022-06-02T12:37:48.376461564Z Downloading numpy-1.22.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
2022-06-02T12:37:48.814161879Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 MB 29.7 MB/s eta 0:00:00
2022-06-02T12:37:52.537935934Z Installing collected packages: numpy
...
(previously I named the file post-install.sh instead of post-init.sh and that’s why it didn’t work… 
)
Thank you, that did it! 