How to deploy Streamlit in renku

How to deploy Streamlit in renku

Streamlit is a great package to create dashboards from python scripts. It is straight-forward to implement and fast to learn. Just import streamlit in your python script and add the streamlit code snippets you need to interact with your project. The resulting dashboards look clean and simple, displaying only what you want to show in a well-designed manner.

The main concepts of streamlit are nicely explained here: Main concepts<!-- --> - Streamlit Docs
And the API reference provides a great overview of streamlitā€™s functionality: API Reference<!-- --> - Streamlit Docs

Streamlit is open-source and can be easily used with your renkulab project in a few steps.

In your renku repository

To deploy a streamlit app with renku, carry out the following steps. If you donā€™t have a renku project yet, see how to get started here.

  1. add these libraries to requirements.txt

    • jupyter-server-proxy
    • streamlit
  2. tell jupyter how to start the streamlit app

    • add jupyter_notebook_config.py to the main path of your repo (see below)
    • add this line to the Dockerfile: COPY jupyter_notebook_config.py ${HOME}/.jupyter/
  3. set the defaults for the interactive environment in .renku/renku.ini

jupyter_notebook_config.py

The configuration to start the app is set in this file. The name of the streamlit environment will be used by renku to reference it.

# Configuration file for jupyter-notebook.

c.ServerProxy.servers = {
    'streamlit': {           # the name of the environment
        'command': [
            # the command to start the streamlit app
            'streamlit',
            'run',
            'app.py',
            '--server.port', '8501',
            '--browser.serverAddress', '0.0.0.0',
        ],
        'port': 8501,
        'timeout': 60
    }
}

Make sure to match the streamlit server port and the jupyter port. This is especially important if you want to open several apps in one session (see below).

.renku/renku.ini

Add the following few lines to your renku.ini file. You might need to create the file, if it doesnā€™t exist yet.
Make sure to use the same name for the streamlit environment as in the jupyter notebook configuration file, preceded by a slash.

[renku "interactive"]
# again the name of the environment for renku
default_url = /streamlit
lfs_auto_fetch = true
cpu_request = 1
mem_request = 2G

If the app works with data in git lfs, set lfs_auto_fetch to true. The values for the number of CPUs and amount of memory should be set depending on the needs of your app.

Note: In this file, comments need to be on separate lines. Otherwise, they are interpreted as part of the argument value.

On renkulab

When you now want to start a new interactive renku environment, you get a third default environment named /streamlit.

Also, the defaults for number of CPUs, amount of memory and the flag to automatically fetch all git lfs data are adjusted.

And now, you can start an environment and when you connect to it, it automatically opens the streamlit dashboard.

Enhancement

Add several apps in one interactive environment

Currently, you can only add one additional default environment to the renku UI.
A simple work-around is to have a main app, which is accessed from the renku UI and from which other apps can be opened.

Adjust jupyter_notebook_config.py such that each app gets started on a separate port, e.g.

# Configuration file for jupyter-notebook.

c.ServerProxy.servers = {
    'streamlit': {           # the name of the main environment
        'command': [
            'streamlit',
            'run',
            'app.py',
            '--server.port', '8501',
            '--browser.serverAddress', '0.0.0.0',
        ],
        'port': 8501,
        'timeout': 60
    },
    'streamlit2': {           # the name of the second environment
        'command': [
            'streamlit',
            'run',
            'app2.py',
            '--server.port', '8502',
            '--browser.serverAddress', '0.0.0.0',
        ],
        'port': 8502,
        'timeout': 60
    }
}

HTML links can be to added your app.py file using st.markdown(). You need to tell streamlit to allow html by setting the flag unsafe_allow_html to True.

st.markdown(
    "<a href='../streamlit2'>the second streamlit app</a>",
    unsafe_allow_html = True
)

If you want to add the additional app in a new tab, use the following:

st.markdown(
    "<a target='_blank' rel='noopener noreferrer' href='../streamlit2'>the second streamlit app</a>",
    unsafe_allow_html = True
)
5 Likes

Very interesting Lilian. Thanks. Would be great to see an R Shiny equivalent of that (if not already around)

2 Likes

Hi @marounbs, this is in fact already possible, with a little bit of additional tweaking! I made a small example project to demonstrate how to set it up.

I donā€™t have any simple shiny apps to add to this demo, would be great if you could contribute one! Do you think it would make sense for us to include this in the default R/RStudio setup?

2 Likes

Thanks @rrrrrok . I will try to look into it when I can and see if I can add a simple app. Also, had a discussion with @cchoirat, who has a lot of better insights than me, so I guess the Shiny topic has been already visited at SDSC and itā€™s only a matter of finding the solution that makes sense to the most users. From my perspective, having this as an ā€œenvironmentā€ by default would be interesting, and I donā€™t think this would add much complexity to the existing template.

@marounbs if you could add just a super simple demo app I would be very grateful!

Regarding adding this per default - Iā€™ll have to check how much bigger it makes the resulting image. If the difference is small then I agree that itā€™s not a problem to add it. Otherwise Iā€™d prefer to leave it out by default but with a simple path for users to add it with minimal effort.

Speaking of templates - @LiliGasser we should turn the Streamlit setup into a template! Maybe we could have a collection of ā€œunofficialā€ community-built templates for special use-cases like this one?

@rrrrrok I like your idea! Having a streamlit template for renku would be very convenient.

Ok great, we can work on preparing a repo that would be a place for such templates. What I have in mind is to set up something where itā€™s easy for others to add new useful project templates and we can help them along to make sure they are fully functional and well documented etc. In the new renkulab (not yet on renkulab.io but you can see it on dev.renku.ch) we allow users to specify a different source of templates, so that would be the natural thing to do IMO.

@LiliGasser Would it be possible to ā€œshareā€ Streamlit dashboards in Renku as you can with Github repositories at https://share.streamlit.io ?

The most obvious solution is to start Interactive Environments as an anonymous user like with @rrrrrokā€™s Shiny example.

But do you think we can have some sort of publicly visible and automatically launched interactive deployments with an associated timeout ? I noticed that there was also a timeout specified in the Community-contributed project templates for Streamlit.

Thanks!

Hi @champost - if I understand share.streamlit.io correctly, it works specifically with GitHub - so if you really wanted to do that, you could mirror your repo on GitHub? otoh anyone can launch a new environment on a public project on renkulab - so if your repo is public, the streamlit app can be used by anyone.

But maybe you have in mind some sort of page that would automatically have the streamlit app running all the time for different anonymous users - enabling something like this is something weā€™ve discussed at various points but we donā€™t really have a timeline for such a feature atm.

1 Like

Thanks @firat for pointing out, that some of the options in renku.ini are not mentioned in the initial description.

If someone is looking for how to request storage: disk_request

Here is the link to all the options: Templates in Renku ā€” Renku documentation

It can also be done through the UI if one is a maintainer of the project (on the renku project page under Settings/Sessions)

(This reply is a compilation of suggestions from @rrrrrok @ralf.grubenmann @cramakri)

1 Like

Hi @rrrrrok thanks for the Shiny app deployment template I was just trying it out with a fairly simple Shiny App that I have setup as an R package. I was wondering if you could help me out with a question about how to use it?

here is my project: Renku

Iā€™m not sure about the last step of getting the session that is launched to actually point at the running app it looks from the logs and the landing page I get when I start a session that the shiny server is running but Iā€™m not sure of the next steps.

I tried adding to the dockerfile as one might when ordinarily making a dockerised shiny app:

EXPOSE 3838
CMD R -e 'options("shiny.port"=3838,shiny.host="0.0.0.0");MutantSets::launchApp()'

But iā€™m not sure if this is perhaps just something with the ports or if maybe Iā€™m missing something about the way this interacts with jupyter.

logs:

2022-05-12T08:37:13.790649322Z [I 08:37:13.789 NotebookApp] Serving notebooks from local directory: /home/rstudio/work/mutantsets
2022-05-12T08:37:13.790711763Z [I 08:37:13.790 NotebookApp] Jupyter Notebook 6.4.5 is running at:
2022-05-12T08:37:13.790726229Z [I 08:37:13.790 NotebookApp] http://racton-40u-mutantsets-8e8eba71-0:8888/sessions/racton-40u-mutantsets-8e8eba71/
2022-05-12T08:37:13.790771952Z [I 08:37:13.790 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2022-05-12T08:37:13.797493554Z [W 08:37:13.796 NotebookApp] No web browser found: could not locate runnable browser.
2022-05-12T08:37:39.253798181Z [I 08:37:39.253 NotebookApp] 302 GET /sessions/racton-40u-mutantsets-8e8eba71 (127.0.0.1) 1.810000ms
2022-05-12T08:37:39.280314266Z [I 08:37:39.279 NotebookApp] 302 GET /sessions/racton-40u-mutantsets-8e8eba71/shiny? (127.0.0.1) 2.000000ms
2022-05-12T08:37:39.763822228Z [2022-05-12T08:37:39.761] [INFO] shiny-server - Shiny Server v1.5.18.979 (Node.js v12.22.6)
2022-05-12T08:37:39.764231896Z [2022-05-12T08:37:39.763] [INFO] shiny-server - Using config file "/tmp/tmpyn1wyw93"
2022-05-12T08:37:39.809634921Z [2022-05-12T08:37:39.809] [INFO] shiny-server - Starting listener on http://[::]:49155
2022-05-12T08:38:30.848408077Z [I 08:38:30.847 NotebookApp] 302 GET /sessions/racton-40u-mutantsets-8e8eba71 (127.0.0.1) 1.640000ms
2022-05-12T08:38:30.913257300Z [I 08:38:30.912 NotebookApp] 302 GET /sessions/racton-40u-mutantsets-8e8eba71/shiny? (127.0.0.1) 1.700000ms
1 Like

Hi @RichardJActon - I believe the shiny server is working as expected, but there is no app that has been built yet? When I launch your project and open a session I see this:

image

which I think is the shiny server just showing the local directory listing since no app has been made yet.

Thanks for the quick response @rrrrrok. Itā€™s been a bit since I last build a shiny app into a docker image but If Iā€™m not forgetting a step this:

EXPOSE 3838
CMD R -e 'options("shiny.port"=3838,shiny.host="0.0.0.0");MutantSets::launchApp()'

Should be all thatā€™s needed for the app to be built and Run. Installing MutantSets from the repo GitHub - CECADBioinformaticsCoreFacility/MutantSets: A tool for selecting candidate mutations from *C. elegans* mutagenesis screens. designed to work with the output of this Galaxy pipeline: https://usegalaxy.eu/u/richardjacton/w/c-elegans-ems-mutagenesis-mutation-caller and running it with MutantSets::launchApp() in a local session start the app just fine.

Hi Richard,

The way you launched the Shiny app would work if you had app.R in the working directory. Otherwise you can do this:
Peek 2022-05-12 11-55

You can set the default endpoint in .renku/renku.ini to be rstudio as a default.

Let us know if that works for you.

Gavin

Ah so itā€™s a path thing, Thanks @gavin-k-lee - I think there is a step to copy the contents of inst from the installed package to the /srv/shiny-server directory that I missed as there is a dummy app.r file in inst for this purpose: MutantSets/app.R at master Ā· CECADBioinformaticsCoreFacility/MutantSets Ā· GitHub I should just be able to copy this file to the project working directory instead using system.file instead then. Iā€™ll see if that works.

That Fixed it!

Just made this change:

EXPOSE 3838
CMD R -e 'options("shiny.port"=3838,shiny.host="0.0.0.0");file.copy(system.file("shinyApp/app.R", package = "MutantSets"),"/home/rstudio/work/mutantsets/app.R")'

Thanks @gavin-k-lee & @rrrrrok Would you like me to add something about this to your example repo @rrrrrok?

1 Like

Thanks @RichardJActon - I think youā€™re almost there :slight_smile:

The solution you have above will work in a local docker container but not when itā€™s running on renkulab because a) we override the command that is executed in the container and b) we mount the repository in that directory in a different container. The safer way to do this (imo) is to use a post-install script that copies over the file, since that runs after the container starts.

Iā€™ve done that in my fork of your project - notice there is a post-install.sh file and an added line to .gitignore that tells git to ignore app.R. Now if you start a session from my project you will drop into the shiny app.

Ah OK good to know, Thanks!

Hi @rrrrrok I was trying to make a version of your shiny-example repository that was itself a minimal shiny app that would run when the example repo is launched.

I made the example repo into a shiny app in an R package and added a line to the post-init.sh that should install the R package then copy the app.R file to the project home directory as I previously did successfully for a shiny app package installed from an external source in install.R

#!/bin/sh
R -e 'remotes::install_local("/home/rstudio/work/shiny-example/", force = TRUE, dependencies = TRUE, upgrade = "never")'
R -e 'file.copy(system.file("shinyApp/app.R", package = "exampleApp"), "/home/rstudio/work/shiny-example/app.R")'

I wanted to do the install_local after the docker image built in the post script so that I would not necessarily need to rebuild the image to run an updated version of the app. Iā€™ve not tried doing the local install during the image build yet - need to give this a try.

The app.R file does not seem to get copied to the project directory during the startup process as it does not appear in the list of files in the Jupyter index so failure seems to be at or before the file copy but Iā€™m not sure how and not certain where the logs from the post-init.sh (if any) might be located to try and debug it. I took a look in the Jupyter Server logs after starting the image and there didnā€™t seem to be anything relevant in there:

2022-08-22T08:20:57.748064954Z /home/rstudio//.Rproj
2022-08-22T08:20:57.843705707Z chown: changing ownership of '/home/rstudio/.rstudio/projects_settings/next-session-project': Operation not permitted
2022-08-22T08:20:57.843777292Z chown: changing ownership of '/home/rstudio/.rstudio/projects_settings': Operation not permitted
2022-08-22T08:20:57.913592983Z fatal: repository '@//renku-env.git' does not exist
2022-08-22T08:20:57.981127072Z find: ā€˜/tmp/renku-envā€™: No such file or directory
2022-08-22T08:20:58.714452143Z [W 08:20:58.714 NotebookApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
2022-08-22T08:20:59.731235233Z [I 08:20:59.729 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
2022-08-22T08:20:59.731310771Z [I 08:20:59.730 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
2022-08-22T08:20:59.852916024Z [I 08:20:59.852 NotebookApp] Serving notebooks from local directory: /home/rstudio/work/shiny-example
2022-08-22T08:20:59.852978972Z [I 08:20:59.852 NotebookApp] The Jupyter Notebook is running at:
2022-08-22T08:20:59.852995165Z [I 08:20:59.852 NotebookApp] http://racton-40u-shiny-2dexample-1f92e9fb-0:8888/sessions/racton-40u-shiny-2dexample-1f92e9fb/
2022-08-22T08:20:59.853009084Z [I 08:20:59.852 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
2022-08-22T08:20:59.858340556Z [W 08:20:59.857 NotebookApp] No web browser found: could not locate runnable browser.

Running these commands work from an interactive RStudio session to launch the app but the launch directly to shiny server is not.

Do you or anyone else have any Ideas where I should look next to get this working?

Hi Richard,

Unfortunately I am a bit confused as to your question. What high level goal are you trying to achieve? In your repository I see 2 app.R files and Iā€™m not sure which you are trying to run.

Gavin