Gurobi license for containers

RenkuLab
1

Hi all

we’re looking into using renku for a lecture using gurobi. They offer licenses that work with containers but seem to be somewhat particular about how to work with them in containers. Is it possible to attach “secret” files to containers on a per session / user basis?

https://hub.docker.com/r/gurobi/optimizer

cheers

2

Hi @a_user, there is currently no straightforward way to attach a file to a session on a per user/session bases.

What does “secret” mean? Can the user who is running that session access the file?

Here is how you can do this:

  • Add the file in the docker image for the session (anyone who has access to the docker image can read the file though).
  • You can add a script called post-init.sh at the root of the project. If present, this is run every time a session is created before the session starts. Here you can download the license file from somewhere and you could potentially download a different license file for every session. This script will have access to the git configuration for the project used by the session. And from the git config you can extract the user’s email and password. And depending on how your renku deployment is setup you will have to handle anonymous users whose username and email will not be set in the git configuration. But you can also just tell your students that they have to be logged in to run sessions. Some renku deployments can fully disable the option to allow not-logged in users to run sessions on public projects.

Note that in both of these cases the user of the session will have access to this license file.

Does one of these options seems like something that may work for you?

3

Hi @tolevski

Thanks for your reply. Secret in the sense that it’s user specific when starting sessions from a shared project. Aka not a shared license file in git. The link to docker hub has a description for kubernetes, but I don’t think that functionality is exposed through renku. The license file itself (or ID in case of an env variable) will be provided by the students individually, so no issue with them accessing it.

If possible it would be great to avoid having to upload a license file on every session start. The expected location is outside of the project folder, though a env variable should work for the path.

We don’t control the license server so getting access to the user credentials on renku wouldn’t help either.

From what I understand here docker-optimizer/PARAMS.md at master · Gurobi/docker-optimizer · GitHub it would be possible to retrieve the license “automatically” but that requires a set of personalised env variables. I don’t think renku has something like that?

cheers

4

@a_user the newest version of renku has the ability to set custom environment variables before starting a session that will be available in the session. So you could set these every time you start a session and have the license in the session.

Is this acceptable?

This has not been deployed anywhere but upgrading to this version is not hard. Which renku deployment are you using? Is it at University of Fribourg? If yes, I think I may have already spoken to a colleague of yours about this.

image
image898×869 70.1 KB

5

Hi @a_user (cc @tolevski),

Indeed we had a request recently at the University of Fribourg for using the Gurobi optimizer in a Julia environment and for a classroom use, which closely resembles your request.

Based on our experience,

However, this project will not work without the gurobi.lic file (free for academic use) which you will have to download from your Gurobi account. As the Gurobi WLS documentation states :

the gurobi.lic file will always contain the following three lines:

WLSACCESSID=203dec48-e3f8-46ac-0184-92d7d6ded944
WLSSECRET=a080cce8-4e01-4e36-955e-61592c5630db
LICENSEID=12127

These give the WLS license ID, the access ID for that license, and the corresponding secret key. These credentials are private and should not be shared with anyone who is not entitled to use your license.

In a running Renkulab session you should upload your own gurobi.lic file to your home folder (a.k.a. /home/jovyan/work/gurobi-wls-project-setup). As the environment variables have already been specified in post-init.sh, the Gurobi optimizer should be available immediately.

You can test the installation of the license from a terminal by running the command gurobi.sh. Simultaneously, you can also check the tokens issued by Gurobi for the containers associated with your gurobi.lic file.

Going forward, you need to commit the changes (addition of the gurobi.lic file) in your own project (clone/fork etc.) if want the changes to be persistent. Also, in keeping with the Gurobi licensing requirements, this project will have to be private. And finally, students can upload their own license files to their Renku sessions/projects for your class.

Hope this helps !

6

@tolevski We’re at eth and haven’t decided yet where it will be run, either renkulab or if there’s going to be an instance available here.

Those env variables look quite interesting, though I can’t tell you right now if they would be all that’s needed. Can they be fixed per user&project (and remembered between sessions) or do you need to put them in repeatedly every time a new session gets started?

@champost Thanks for your input. If I read the docs right setting env variables with these values gurobi should be able to acquire the license automatically without uploading the license file explicitly.

The Docker example here passes these license key-values as env parameters, though I have to check if they rebuild the license file from them or if that works without further intervention.

https://hub.docker.com/r/gurobi/python-example#

$ docker run -e GRB_WLSACCESSID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-e GRB_WLSSECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
-e GRB_LICENSEID=99999
gurobi/python-example

cheers

7

Right now the env vars need to be put in on every session start, though they can be included in a session autostart link. We do want to support more permanent storage of secrets/env vars in the future, but don’t have a fixed timeline on when and as you can imagine, there are a lot of security considerations to make before adding this.

8

@a_user What I provided was a MWE for using Gurobi on Renku. If your students can make private forks of the my project, download and commit their own gurobi.lic licence file (a one time procedure) at the outset of your course then you can concentrate only on your course content for the rest of the semester !

You could also, as Gurobi is free for academic use, commit your gurobi.lic licence file to a private fork and enable access to your students. They would then use tokens issued under your WLS license for the containers they start. You will need to check the upper limit of container sessions for a single WLS license, which I think is quite large.

IMHO, testing Kubernetes/Docker env. vars. within Renku feels a bit excessive (a high effort/low impact task) for your use case.

9

@champost

The project will be a collaborative one between a small group of students each, hence a shared fork, and the need(?) for separate licenses in a project. Though I will check how many tokens can be used for any given license simultaneously and see if that is an acceptable solution.

10

@a_user

I was also searching for it and found the following (difficult to find) information on the possible number of concurrent sessions :

Limits

A license may have a baseline limit expressed with a number of concurrent sessions. A session is a continuous use of Gurobi software in a container or machine. In this case, you are entitled to run concurrent sessions up to the baseline, and also go over the baseline for a short period of time. This means that you can typically operate rolling updates or blue/green deployments where the number of concurrent sessions may temporarily exceed the baseline with a factor 2.

However, if the recorded usage goes over the baseline for too long or by too much, the token requests may be rejected, and your application will not be able to start new optimization. Note that you cannot release a token, even if you are done using Gurobi. This means that if the container or machine is shut down before the token lifespan has been reached, it will still count until its expiration time (5 minutes by default).

For example, if the license baseline is 2 concurrent sessions, you can start 3 or 4 containers/machines to perform a rolling update. However, if this usage continues for several minutes, or if you start even more containers/machines, your application may not be able to start new optimization.

The “default” license baseline seems to be 2 concurrent sessions (at least in my case).

11

@a_user just to summarize this discussion here are you options:

  1. insert environment variables when you launch a session through the UI (but these are not saved when you shut the session down and you have to add them every time you launch a session)
  2. insert environment variables by using a long link that in it contains the environment variables themselves and will launch a new session with them when visited (as far as I know this is not documented anywhere but I can give you more details if you think you would use it)
  3. set the environment variables by adding a post-init.sh script at the root of the project, renku looks for a script named like this when it starts a session and will run it, so doing a bunch of statements like export ENV_VAR1=some_value can be used to set things up
  4. add the files/environment variables/etc as part of the Docker image in every project
  5. add the files/environment variables/etc as part of the Docker image in one project and let students fork this single project to start their own projects

Let me know if you need anything else. Also let me know what you ended up using and how it worked.

12

Hi @tolevski

As there is no persistent and per-user setup we’re finally going for the simple case of setting up a license folder in the project and have students upload their license whenever they start a new session.

On that note: env variables are quite a pita in Rstudio. Do you know a reliable way to modify the “PATH” variable for the rstudio based image? Neither the “ENV GUROBI_HOME /opt/gurobi/linux64” directive in the dockerfile, nor the ’ export PATH=“${PATH}:${GUROBI_HOME}/bin” ', nor “echo PATH=/opt/gurobi:${PATH} >> /usr/local/lib/R/etc/Renviron.site” in post-init.sh survive into the active rstudio/r-session in a new session.