Error: Cannot start session via the notebook service

[champost]

Hello,

I am following the instructions for Connecting to RenkuLab Sessions with SSH (Connect to RenkuLab Sessions with SSH & Local IDE — Renku documentation).

My OpenSSH version :

$ ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n  7 Dec 2017

My Renku CLI version :

$ renku --version
2.5.0

I got past the following steps :

renku login https://renkulab.io
renku clone https://renkulab.io/gitlab/<user.name>/<projectname>

But I am stuck at the following step :

$ renku session start --ssh -p renkulab
Error: Cannot start session via the notebook service because {"error":{"code":2000,"detail":null,"message":"Something went wrong."}}

[tolevski]

@champost what project are you trying to do this in?

[champost]

Hi @tolevski, it is a private project. Here is the link : Reproducible Data Science | Open Research | Renku

[tolevski]

This is a bug in one of our services. I will submit a fix. In the meantime you can launch a session through the website and then connect to it with the cli.

renku session ls -p renkulab will show you the running session for the project you are currently in.

Then you can run renku session open --ssh -p renkulab <session_name> to connect via ssh.

[champost]

@tolevski

This is a bug in one of our services. I will submit a fix. In the meantime you can launch a session through the website and then connect to it with the cli.

So I followed your suggestion and this is what I get :

$ renku session ls -p renkulab
Session champak-2e-resibase-2dv3-50112b88 (running, renkulab)
Started: 2023-08-09T15:05:33+00:00
Url: https://renkulab.io/projects/champak.beeravolureddy/resibase-v3/sessions/show/champak-2e-resibase-2dv3-50112b88
Commit: 6865b8b0acb3109020586f5d125407992451b3ad
Branch: master
SSH enabled: no

Even though SSH enabled is set to no, I tried to open a SSH session :

$ renku session open --ssh -p renkulab champak-2e-resibase-2dv3-50112b88
Warning: Permanently added 'champak-2e-resibase-2dv3-50112b88' (ECDSA) to the list of known hosts.
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535

Is this expected behaviour or am I doing something not right ?

[tolevski]

@champost the reason for this error is most likely because you have not added your public ssh key to the project you are trying to use SSH with.

You can do this in two ways:

1. Manually create a .ssh folder in the project, in it create a file called authorized_keys and in here paste any public SSH keys that should be able to access the session.
2. The renku CLI can do the part from option 1 automatically when you try to launch a session with ssh enabled. The problem is that the part that actually launches the session fails but the part that adds a public key works. So after you run this command in your repo locally renku session start -p renkulab --ssh and after it fails simply push the commit/changes the command created. Then launch a session from the website and you will be able to connect.

Let me know if this works. We have future work planned to make SSH access even more painless.

[champost]

@tolevski

the reason for this error is most likely because you have not added your public ssh key to the project you are trying to use SSH with.

@tolevski now that’s what I thought initially !

@tolevski

The renku CLI can do the part from option 1 automatically when you try to launch a session with ssh enabled.

As you suggest, my SSH public key was automatically created, the ~/.ssh/config file modified accordingly and a commit pushed to Gitlab a while ago, all via the CLI. But I am still unable to connect to a session launched from the website for this project.

I will now try this with a blank project and let you know…

[champost]

– UPDATE –

I created a blank project : launch-ssh-session-from-cli.

I authenticated via CLI :

$ renku login https://renkulab.io
Warning: Cannot log in to git outside a project.
Please grant access to 'renku-cli' in your browser.
If a browser window does not open automatically, go to https://renkulab.io/auth/realms/Renku/device?user_code=DLPB-RGLX
Successfully logged in.

Note the Warning: Cannot log in to git outside a project.. The instructions show on the Renkuab UI differ slightly from the ones here : Connect to RenkuLab Sessions with SSH & Local IDE — Renku documentation. I think one needs to clone the repository and change directory beforehand.

So I cloned the repository

$ renku clone https://gitlab.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli.git
Cloning https://gitlab.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli.git ...
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (45/45), done.
OK
$ 
$ cd launch-ssh-session-from-cli/
$ 
$ renku login https://renkulab.io
Warning: Remote URL will be changed. Do you want to continue (to disable this warning, pass '--yes' or run 'renku config set show_login_warning False')? [y/N]: y
Please grant access to 'renku-cli' in your browser.
If a browser window does not open automatically, go to https://renkulab.io/auth/realms/Renku/device?user_code=XPNP-JMRB
Successfully logged in.
$ 

I got a different warning this time which went away by typing y.

Then I launched a session :

$ renku session start -p renkulab --ssh
Info: Adding SSH public key to project.
Info: Added public key. Changes need to be pushed and remote image built for changes to take effect.
Image registry.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli:fa63077 built successfully.
Error: Cannot start session via the notebook service because {"error":{"code":2000,"detail":null,"message":"Something went wrong."}}

[champost]

– UPDATE CONTINUED –

So I launched a session from the Renkulab website and searched for searched for the session name :

$ renku session ls
Session champak-2e-launch-2dssh-2dsession-2-4127de03 (running, renkulab)
Started: 2023-08-11T11:48:08+00:00
Url: https://renkulab.io/projects/champak.beeravolureddy/launch-ssh-session-from-cli/sessions/show/champak-2e-launch-2dssh-2dsession-2-4127de03
Commit: fa6307769c4ecce4dc686ada72202261c9d211d3
Branch: master
SSH enabled: no

It says SSH enabled: no !

And when I try to open the session anyway I get :

$ renku session open --ssh champak-2e-launch-2dssh-2dsession-2-4127de03
Warning: Permanently added 'champak-2e-launch-2dssh-2dsession-2-4127de03' (ECDSA) to the list of known hosts.
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535

My SSH keys seem to have been added fine : Commits · master · Champak Beeravolu Reddy / launch ssh session from cli

[tolevski]

@champost when you use the renku session open --ssh command it will only use keys and the configuration that is created in ~/.ssh/renku. There you should be able to find a file called after the session name ending in .conf and in that file you will see which keys are going to be used to access the session. Those are the only keys that will be used. So if the public key from that keypair form the config file is not in the session then ssh access will not work through the CLI.

But you can access the session without the help of the CLI. Basically on renkulab.io if your session id is champak-2e-launch-2dssh-2dsession-2-4127de03 then you can run ssh -J jovyan@renkulab.io:2022 jovyan@champak-2e-launch-2dssh-2dsession-2-4127de03 to get access. That will work with the keys you have in ~/.ssh and any keys that the renku CLI has generated in ~/.ssh/renku.

The way to access a session is always the same. The most general command is like this:
ssh -J jovyan@<renku-domain>:2022 jovyan@<session_id>. That is it. The CLI runs this under the hood for you with a few added flags setup. And one of these flags is limiting the keys being used to only specific pairs in .ssh/renku and it ignores stuff in .ssh.

[tolevski]

I opened a few issues to address these things. I will work on them next week.

[tolevski]

But even without the fixes that I will work on next week you can still access the session with a regular ssh command as I mentioned in my longer post above.

[champost]

Hi @tolevski ,

I gave this another try.

This time I deleted my blank project and recreated another on Renkulab with the same name (FYI : Reproducible Data Science | Open Research | Renku).

$ renku clone https://gitlab.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli.git
Cloning https://gitlab.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli.git ...
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (44/44), done.
OK

$ renku login https://renkulab.io
Warning: Cannot log in to git outside a project.
Please grant access to 'renku-cli' in your browser.
If a browser window does not open automatically, go to https://renkulab.io/auth/realms/Renku/device?user_code=NKME-TENF
Successfully logged in.

$ cd launch-ssh-session-from-cli/
$
$ renku session start -p renkulab --ssh
Your system is not set up for SSH connections to Renkulab. Would you like to set it up? [y/N]: y
Info: Generating keys
Info: Writing SSH config
Info: Adding SSH public key to project.
Info: Added public key. Changes need to be pushed and remote image built for changes to take effect.
Error: Please run the renku login command to authenticate with Renku.

This error is strange as I just logged into Renkulab a short while ago with browser-based authentication !
However, an SSH key pair (gitlab.renkulab.io-key.pub) has been generated in ~/.ssh/renku/ along with a 99-gitlab.renkulab.io-jumphost.conf file containing following the jumphost details :

Host jumphost-gitlab.renkulab.io*
    HostName gitlab.renkulab.io
    Port 2022
    User jovyan

So I logged in from the CLI a second time as suggested…

$ renku login https://renkulab.io
Warning: Remote URL will be changed. Do you want to continue (to disable this warning, pass '--yes' or run 'renku config set show_login_warning False')? [y/N]: y
Please grant access to 'renku-cli' in your browser.
If a browser window does not open automatically, go to https://renkulab.io/auth/realms/Renku/device?user_code=ORAA-QTXQ
Successfully logged in.

And then…

$ git push 
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://renkulab.io/repos/champak.beeravolureddy/launch-ssh-session-from-cli.git/info/lfs.locksverify true
Counting objects: 4, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 459 bytes | 459.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
To https://renkulab.io/repos/champak.beeravolureddy/launch-ssh-session-from-cli.git
   01b765e..4a13043  master -> master

$ renku session start -p renkulab --ssh
Your system is not set up for SSH connections to Renkulab. Would you like to set it up? [y/N]: 
Error: Can't run ssh session without setting up Renku SSH support. Run without '--ssh' or run 'renku session ssh-setup'.

The error is once again strange as SSH access was already setup via the commit “Add public SSH key.” and was pushed to Renkulab and I waited until the Docker image was built before starting a session.

$ renku session start -p renkulab --ssh
Your system is not set up for SSH connections to Renkulab. Would you like to set it up? [y/N]: y
Info: Generating keys
Info: Writing SSH config
Info: Adding SSH public key to project.
Info: Added public key. Changes need to be pushed and remote image built for changes to take effect.
Image registry.renkulab.io/champak.beeravolureddy/launch-ssh-session-from-cli:9d03ee1 built successfully.
Session champak-2e-launch-2dssh-2dsession-2-4feaf1a1 successfully started, use 'renku session open --ssh champak-2e-launch-2dssh-2dsession-2-4feaf1a1' or 'ssh champak-2e-launch-2dssh-2dsession-2-4feaf1a1' to connect to it

Now this part is where it gets weirder as different SSH key pair (renkulab.io-key.pub) is generated in ~/.ssh/renku/ along with a different .conf file (99-renkulab.io-jumphost.conf) containing the following :

Host jumphost-renkulab.io*
    HostName renkulab.io
    Port 2022
    User jovyan

So was the previous remote (jumphost-gitlab.renkulab.io*) only a Gitlab specific host !? This time, a commit containing both the keys to .ssh/authorized_keys is pushed automatically to remote : Commits · master · Champak Beeravolu Reddy / launch ssh session from cli · GitLab

Also, a new .conf file (00-launch-ssh-session-from-cli-champak-2e-launch-2dssh-2dsession-2-4feaf1a1.conf) appears with :

Host champak-2e-launch-2dssh-2dsession-2-4feaf1a1
    HostName champak-2e-launch-2dssh-2dsession-2-4feaf1a1
    RemoteCommand cd work/launch-ssh-session-from-cli/ || true && exec $SHELL --login
    RequestTTY yes
    ServerAliveInterval 15
    ServerAliveCountMax 3
    ProxyJump  jumphost-renkulab.io
    IdentityFile /home/champost/.ssh/renku/renkulab.io-key
    User jovyan
    StrictHostKeyChecking no

And finally when I attempt to connect wither way (with or without CLI) to the running session :

$ renku session ls
Session champak-2e-launch-2dssh-2dsession-2-4feaf1a1 (running, renkulab)
Started: 2023-08-22T12:40:54+00:00
Url: https://renkulab.io/projects/champak.beeravolureddy/launch-ssh-session-from-cli/sessions/show/champak-2e-launch-2dssh-2dsession-2-4feaf1a1
Commit: 9d03ee16a9c8cab33b92f784c3a8ea3ea1046278
Branch: master
SSH enabled: yes
$
$ renku session open --ssh champak-2e-launch-2dssh-2dsession-2-4feaf1a1
Warning: Permanently added 'champak-2e-launch-2dssh-2dsession-2-4feaf1a1' (ECDSA) to the list of known hosts.
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535
$
$ ssh -J jovyan@renkulab.io:2022 jovyan@champak-2e-launch-2dssh-2dsession-2-4feaf1a1
Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
Disconnected from UNKNOWN port 65535
$
$ renku session ssh-setup
Keys already configured for host renkulab.io. Overwrite? [y/N]: 
Aborted!
$

Voilà !