# Renku migrate hangs

**URL:** <https://renku.discourse.group/t/renku-migrate-hangs/226>\
**Category:** Renku (CLI)\
**Created:** [17 December 2020 13:34 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226 "2020-12-17T13:34:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [17 December 2020 13:34 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/1 "2020-12-17T13:34:34Z")

</div>

I am trying to migrate a repo to a new version of Renku, but it hangs at the second step:

Applying migration m\_0005\_\_1\_pyld2…  
Applying migration m\_0005\_\_2\_cwl…

This happens for this repo:

> **[Renku](https://renkulab.io/projects/remko.nijzink/vomcases)**

The repo contains quite some data, and I think there is something wrong with the metadata, because it can’t create the knowledge graph as well.

---

<div class="post-metadata">

**Author:** ![cramakri](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/cramakri/32/7_2.png) [@cramakri](https://renku.discourse.group/u/cramakri)\
**Post date:** [17 December 2020 13:44 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/2 "2020-12-17T13:44:16Z")

</div>

Everyone who could help with this is on vacation until the new year. Sorry about that, but I will make a reminder to have someone get back to you.

---

<div class="post-metadata">

**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\
**Post date:** [18 December 2020 16:25 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/3 "2020-12-18T16:25:48Z")

</div>

We complete changed our metadata format that we use to store workflows, to enable us to start improving the way we use workflows, but also to improve performance. But since the old format and way of doing things is rather slow, that migration can take quite some time. If you check the number of `.cwl` files in the `.renku/workflows` folder, the time the migration takes is on the order of that number of files times 3 seconds per file (so 200 workflows takes around 10 minutes), though it doesn’t scale linearly and depends on how those workflows are related/connected to each other, how many files the use as inputs/outputs and how complicated the git history of the project is.

If you check out the repo locally and run `renku migrate` manually, you should be able to see it making commits for the workflows while it’s migrating (in a second terminal, for instance).

Just to make sure it’s actually hanging and not just taking a long time.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [18 December 2020 16:54 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/4 "2020-12-18T16:54:29Z")

</div>

Okay, thanks! That sounds actually plausible… I have 9735 files there… so should be at least 8 hours (I gave up after 4 so far). Will try again. But I did it locally anyways, but there were no new commits after I stopped the migration. Should I be able to see that afterwards? Or how can I check if it is actually committing during the migration exactly?

---

<div class="post-metadata">

**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\
**Post date:** [18 December 2020 16:57 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/5 "2020-12-18T16:57:39Z")

</div>

You should see the commits during the migration, due to the way things work it has to make a new commit for each workflow file as it is processed, to not break things.

On a side note, we are working on a way to store all metadata in a single file so all the walking the commit history is not necessary anymore, which speeds things up a lot. This is already done as an experimental feature but will hopefully be fully released in the next 3 months.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [18 December 2020 17:00 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/6 "2020-12-18T17:00:31Z")

</div>

Okay, but I just saw this:

> [@rcnijzink](#):
>
> Applying migration m\_0005\_\_1\_pyld2…  
> Applying migration m\_0005\_\_2\_cwl…

and no new commits afterwards…

---

<div class="post-metadata">

**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\
**Post date:** [18 December 2020 17:02 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/7 "2020-12-18T17:02:45Z")

</div>

Oh I think we misunderstood each other. I meant, run the migration in one terminal and then go to the project folder in a second terminal and do a `git log`, there you should see it creating commits.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [18 December 2020 17:06 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/8 "2020-12-18T17:06:57Z")

</div>

Yes, but so I see no new commits when I do that unfortunately.

---

<div class="post-metadata">

**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\
**Post date:** [18 December 2020 17:37 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/9 "2020-12-18T17:37:21Z")

</div>

That’s weird. I’ll take a look at the project after my vacation and let you know if I find anything.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [13 January 2021 14:09 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/10 "2021-01-13T14:09:28Z")

</div>

I didn’t have problems before, but when I try to run a renku command now I get an error, guess because of the interrupted migration:

Error: Project version is outdated and a migration is required.  
Run `renku migrate` command to fix the issue.

---

<div class="post-metadata">

**Author:** ![mohammad-sdsc](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/mohammad-sdsc/32/79_2.png) [@mohammad-sdsc](https://renku.discourse.group/u/mohammad-sdsc)\
**Post date:** [13 January 2021 14:53 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/11 "2021-01-13T14:53:32Z")

</div>

Hi Remko,

This probably happens because of a renku update in your environment or local machine. Newer versions of Renku won’t work with a repository that was created/modified with an older version due to changes in metadata. That’s why a migration is needed to bring the metadata up to state.

You either have to migrate the project or install a version of renku that would work with your repo. From the above messages I believe that your project’s metadata version is 4 which would work with Renku `v0.10.4`. You can install this version locally by running `pip install renku==v0.10.4`.

Note that if the migration was interrupted and there are migration commits (which does not seem to be the case) then you must either finish the migration or `git reset --hard` to the last commit before migration.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [13 January 2021 15:10 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/12 "2021-01-13T15:10:52Z")

</div>

Yes, thanks! I would prefer to use a new version of Renku, but so the migration still hangs at the second step.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [15 January 2021 17:24 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/13 "2021-01-15T17:24:30Z")

</div>

I ran renku migrate now for a long time, strace showed it was actually doing something. Eventually, I received this error:

```
Applying migration m_0005__1_pyld2...
Applying migration m_0005__2_cwl...
You are using renku version 0.12.2, however version 0.12.3 is available.
You should consider upgrading ...
Error: Couldn't execute migration

Traceback (most recent call last):
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrate.py", line 128, in migrate
    module.migrate(client)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrations/m_0005__2_cwl.py", line 48, in migrate
    _migrate_old_workflows(client)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrations/m_0005__2_cwl.py", line 87, in _migrate_old_workflows
    path = _migrate_cwl(client, cwl_file, commit)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrations/m_0005__2_cwl.py", line 107, in _migrate_cwl
    _, path = _migrate_single_step(client, workflow, path, commit=commit, persist=True)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrations/m_0005__2_cwl.py", line 206, in _migrate_single_step
    matched_input = next(i for i in inputs if i.id == name)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/exception_handler.py", line 87, in main
    return super().main(*args, **kwargs)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/cli/migrate.py", line 61, in migrate
    skip_template_update=True, skip_docker_update=True, progress_callback=click.secho,
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/commands/client.py", line 90, in new_func
    raise result.error
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/incubation/command.py", line 128, in execute
    output = context["click_context"].invoke(self._operation, context["client"], *args, **kwargs)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/commands/migrate.py", line 78, in migrate_project
    progress_callback=progress_callback,
  File "/home/remko/.local/pipx/venvs/renku/lib/python3.6/site-packages/renku/core/management/migrate.py", line 130, in migrate
    raise MigrationError("Couldn't execute migration") from e
renku.core.errors.MigrationError: Couldn't execute migration
```

---

<div class="post-metadata">

**Author:** ![mohammad-sdsc](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/mohammad-sdsc/32/79_2.png) [@mohammad-sdsc](https://renku.discourse.group/u/mohammad-sdsc)\
**Post date:** [1 February 2021 08:42 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/14 "2021-02-01T08:42:10Z")

</div>

We’ve made a new Renku CLI release, `v0.13.0`, that fixes migration errors you had in this project. It also shows migration’s progress. You need to hard reset the project to the last commit before migration and remove any extra files that is there (be careful with these commands as they cannot be undone):

```auto
git reset --hard <commit-sha>
git clean -dfx

```

Please upgrade to this version and try migrating again.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [16 February 2021 09:50 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/15 "2021-02-16T09:50:59Z")

</div>

Thanks! It finally worked, but it took a really long-time… It had to run for 11 days (!) before it finished. Now I just have 9736 new commits, and when I try to push, it says:  
` Connection to renkulab.io closed by remote host.`

Is that just my internet connection, or do I have too many commits or so?

---

<div class="post-metadata">

**Author:** ![pameladelgado](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/pameladelgado/32/28_2.png) [@pameladelgado](https://renku.discourse.group/u/pameladelgado)\
**Post date:** [17 February 2021 10:57 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/16 "2021-02-17T10:57:01Z")

</div>

@rcnijzink this looks like an issue with having too many commits and a connection timeout.  
May I ask if you are using `ssh` or `https` for the push? (one way to check this is with `git remote show origin`)  
If the origin is an `https` link, could you try using `ssh` instead?

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [17 February 2021 12:08 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/17 "2021-02-17T12:08:43Z")

</div>

Yes, I think it is something like that. I am using ssh here, is there a way I can change the timeout time?

---

<div class="post-metadata">

**Author:** ![pameladelgado](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/pameladelgado/32/28_2.png) [@pameladelgado](https://renku.discourse.group/u/pameladelgado)\
**Post date:** [17 February 2021 12:31 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/18 "2021-02-17T12:31:50Z")

</div>

We could try modifying your ssh settings in the `~/.ssh/config` file to keep the connection alive, as described [here](https://stackoverflow.com/questions/60833006/connection-to-github-com-closed-by-remote-host-when-pushing).  
The following will send a null packet to the server every 60 seconds (`ServerAliveInterval`) for 30 times (`ServerAliveCountMax`) leaving the connection be alive for 30 minutes in total. If that’s not enough you can increase `ServerAliveCountMax`.

```auto
Host *
  ServerAliveInterval 60
  ServerAliveCountMax 30

```

You could also prepend this `GIT_SSH_COMMAND="ssh -vvv"` to your `git push` command to get a more verbose output.

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [18 February 2021 09:06 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/19 "2021-02-18T09:06:33Z")

</div>

Thanks! Unfortunately it still doesn’t push, but it also seems as if the connections is not longer than 60 seconds alive. This is the last part of ` GIT_SSH_COMMAND="ssh -vvv" git push` :

```
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 5/6 cc -1)

debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Connection to renkulab.io closed by remote host.
Transferred: sent 4020, received 3564 bytes, in 60.2 seconds
Bytes per second: sent 66.7, received 59.2
debug1: Exit status -1
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://renkulab.io/remko.nijzink/vomcases.git/info/lfs.locksverify true
```

---

<div class="post-metadata">

**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\
**Post date:** [19 February 2021 15:20 UTC](https://renku.discourse.group/t/renku-migrate-hangs/226/20 "2021-02-19T15:20:39Z")

</div>

I managed, but maybe not in the most elegant way…

`git log -9736 --pretty=oneline > ../log`

Then I removed all the commit messaged so it was just a file with the commit number on each line.

Then reverted:

`tac ../log > ../log_reverted`

And had a simple script to push each commit:

```
for commit in $(cat ../log_reverted)
do

git push origin $commit:master

done
```

[Next page](https://renku.discourse.group/t/renku-migrate-hangs/226.md?page=2)