Very slow image download recently?

Hello!

It seems to me that starting session got much slower in the recent weeks/months. The delay is at “Downloading image” stage.

For “small” images I wait many minutes (used to be 10s of seconds).

For 4Gb image, it took about an hour I think, this one Reproducible Data Science | Open Research | Renku

I understand that once it is pulled, next session might be faster if it is on the same node or something like that. But, an hour seems a bit long, doesn’t it?..

Cheers

Volodymyr

Hi @volodymyrss - thanks for reporting this - we haven’t noticed anything extraordinary recently that could account for this. I tried launching a session from your project and it took about 7 minutes:

Successfully pulled image "registry.renkulab.io/saasfee2022/course-materials:4f412c0" in 6m23.113913556s

It could be that you happened to try at a particularly busy time? In any case, we’ll have a look and try to track it down but please let us know if you continue experiencing this.

Hi @rrrrrok

Thanks for the feedback!

Yes, I acknowledge that right now it starts pretty quickly, no problem!

It’s not possible to track on your side this “overuse” event when it caused me to wait for so much longer?
I understand that in principle there is no guarantee it will not take 1h at some other point.

I want to show this case to some interested very colleagues, it’s just one session. Or maybe (hopefully!) they will rush to experiment themselves immediately.
But if it takes 1hr to start it will be unfortunate.
To mitigate possible issue like that, I will start the session in advance this time.

Cheers

Volodymyr

I see a launch that took ~30 minutes on that project at around 9:45pm on the 28th (i.e. 2 days ago) - does that sound like the correct event?

Sounds like that’s it! I was in IC1 Tuesday evening and I think it took a bit more than from about Geneva to Lausanne, but that includes me coming back to the session.
It’s in principle possible it is about 30 min, less than I thought, but still preventing interactive demonstration.
So was it because of the busy time?

Unfortunately there’s not much we can do about your commute time :smiley:

No it was not very busy then - not sure what the reason was. We’ll dig into it a bit more, but with large images like that it’s probably better to start it ahead of time anyway, even a few minutes during a demo is annoyingly long.

Ok, noted.
I does make me wonder how is that a train managed to me half way along the lake, while some 4Gb were moving from the registry (?) to a compute node, is it really 2Mb/s? It did seem to be waiting on downloading.

There a number of reasons why it could take longer than expected, besides slow network speed. We’ll try to understand if anything under our control is to blame…

1 Like

Just another observation, positive, just now (and once earlier today) for me, the same image got downloaded much quicker than 6min you quote, just few seconds (really). Hope this helps!