Disconnected from the server while working on a project

Hello Renku team,

I was working on a project on the SV-renku deployment when I got disconnected from the environment.

My environment is now inaccessible with the yellow loading sign running permanently.

The problem persists upon disconnecting / reconnecting from renkulab and caused me to lose an hour of work.

It could well be that the problem comes from a local issue at EPFL, but just in case I’m pasting the end of the log file below.

Best,

Cyril

[D 2021-04-08 13:39:40.434 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[D 2021-04-08 13:39:40.435 SingleUserNotebookApp init:95] NBResuse metrics: {‘rss’: 697118720, ‘limits’: {‘memory’: {‘rss’: 1073741824, ‘warn’: False}}}
[I 2021-04-08 13:39:40.436 SingleUserNotebookApp log:174] 200 GET /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/metrics?1617889180397 (pulver@10.244.1.0) 2.61ms
[D 2021-04-08 13:39:41.049 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:41.049 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[D 2021-04-08 13:39:41.050 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:41.050 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[I 2021-04-08 13:39:41.053 SingleUserNotebookApp log:174] 200 GET /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/api/contents/data/temp?content=1&1617889181019 (pulver@10.244.1.0) 5.29ms
[I 2021-04-08 13:39:41.054 SingleUserNotebookApp log:174] 200 GET /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/api/terminals?1617889181017 (pulver@10.244.1.0) 4.64ms
[D 2021-04-08 13:39:41.758 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:41.758 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[I 2021-04-08 13:39:41.778 SingleUserNotebookApp log:174] 200 POST /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/git/branch?1617889181725 (pulver@10.244.1.0) 20.31ms
[D 2021-04-08 13:39:41.791 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:41.791 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[I 2021-04-08 13:39:41.813 SingleUserNotebookApp log:174] 200 POST /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/git/status?1617889181767 (pulver@10.244.1.0) 23.24ms
[D 2021-04-08 13:39:42.411 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:42.411 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
[D 2021-04-08 13:39:42.412 SingleUserNotebookApp init:95] NBResuse metrics: {‘rss’: 697118720, ‘limits’: {‘memory’: {‘rss’: 1073741824, ‘warn’: False}}}
[I 2021-04-08 13:39:42.413 SingleUserNotebookApp log:174] 200 GET /jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/metrics?1617889182374 (pulver@10.244.1.0) 3.01ms
[D 2021-04-08 13:39:43.833 SingleUserNotebookApp auth:871] Allowing whitelisted Hub user pulver
[D 2021-04-08 13:39:43.833 SingleUserNotebookApp auth:744] Setting oauth cookie for 10.244.1.0: jupyterhub-user-pulver-promoter-te-subfamilies-matrix-00d60a51, {‘path’: ‘/jupyterhub/user/pulver/promoter-te-subfamilies-matrix-00d60a51/’, ‘httponly’: True, ‘secure’: True}
/entrypoint.sh: line 31: 25 Killed $@

Hello @bopekno

Indeed your environment restarted, at the moment I don’t have more information about the reason (it could be some local infrastructure issue as you mentioned, notably nodes running out of resources).

As for the yellow state of the environment, there is a related fix on recent Renku versions that hasn’t been yet deployed on SV-renku. However, this would not have prevented losing unsaved work. For the moment I have cleaned up this environment so that you can start a new one.

We are working hard on making Renku more resilient to these kind of events, but these changes will take some time to reach production. Was the work maybe saved in a temporary autosave branch?

Best,
Pamela

1 Like

Hi @bopekno

I just wanted to let you know that the SV-renku deployment has just been updated to the latest version. Now you can enjoy having the latest new features, improvements and fixes in that deployment as well.

Best,
Pamela