How to move projects between different RenkuLab Deployments

FAQ
1

We have users interacting with two or more Renkulab deployments, and it’s occasionally needed to move a project between them.
Here is a recap of what you should do. The explanation is rather verbose, but it’s actually only a few commands.

Depending on a few factors (e.g. amount of data in the project, accessibility of the deployments, etc.), it’s possible to choose between 2 alternatives:

  1. Start an Interactive Environment on the source deployment.
  2. Clone the project locally and push it.

Depending on which technique you choose, the first steps are different.

Case 1 - Interactive Environment

  • Start a new interactive environment in the project you want to migrate. Optional: you can turn on the option to Automatically fetch LFS data. Depending on the amount of LFS data, that may extend the startup time.

  • Verify that the lfs files are checked out properly by typing

    git lfs pull

    If anything is missing, it will be downloaded.

Case 2 - Local environment

  • Verify you have both git and git lfs installed locally
    git lfs
  • Log in the source deployment, go to the project you want to clone, and select the “Settings” tab. Copy and execute the clone command locally in an empty folder. TIP: If you have the renku package installed in your command line, you should prefer the renku clone solution over git clone.
    This step may require some time, depending on the amount of data and your internet connections.

Common steps

  • It’s possible to push the content directly to the target deployment without first creating an empty project. It’s important to push to a valid combination of namespace/project name though.
    The target url will have the following structure
    https://<deployment_gitlab_url>/<namespace>/<project_name>.
    How do you find the correct values?

    • You can get the deployment_gitlab_url by connecting to the target deployment and clicking on the GitLab icon on the top right. This will open the GitLab instance. Copy the URL in the address bar of your browser.
      TIP: the standard URL is usually the base RenkuLab URL plus /gitlab. For renkulab.io, it’s Reproducible Data Science | Open Research | Renku
    • The namespace can be chosen between your groups and your default user namespace. You can find it by clicking on “New project” and checking the “Namespace” section.
      TIP: each user should have at least a namespace corresponding to the username.
    • The project_name can be any valid name that is not already taken in the target namespace. There are quite a few restrictions on the naming of a project. You can easily get a valid name and verify that it’s not already taken by going to the “New project” page and inserting the desired name in the “Title” field. You will get an escaped version of the title in the “Identifier” field.
      TIP: copy the whole “Identifier” files to get a valid combination of <namespace>/<project_name>.

    The final URL should be something like this: Reproducible Data Science | Open Research | Renku

  • OPTIONAL: if you need to push all the branches, you need to track and fetch them

    for i in `git branch -a | grep remote | grep -v HEAD | grep -v master`; do git branch --track ${i#remotes/origin/} $i; done`

    then you have to pull the branches

    git pull --all

    You can verify that all the branches are cloned by executing

    git branch -a

    TIP: should you have any stale or unneeded branch, this is a good time to get rid of it.

  • Add the target URL as a new remote:

    git remote add target https://renkulab.io/gitlab/fake-user/non-existing-project

  • Push the content to the new remote. If you need to push only the current branch, let’s say master, you can execute:

    git push -u target master

    You will be asked for your credentials.

    Problems with authentication?
    The GitLab credentials may be different from the RenkuLab credentials. That is generally the case if you created your account through a third party (E.G: GitHub).
    If you get an authentication error while pushing, verify on the GitLab profile page that you are using the correct password and you don’t see any warning at the top of the page (try from the password edit page, E.G. Reproducible Data Science | Open Research | Renku)
    If you still have problems, you can push using an access token. Create a new one on the Access Token page (E.G. Reproducible Data Science | Open Research | Renku) with the write_repository scope included, and use it in your target URL in the form
    https://oauth2:<access_token>@<git_url>
    (E.G. https://oauth2:bXiMy4UvTa@renkulab.io/fake-user/non-existing-project).

    If you need to push all the branches, execute

    git push target --all

    TIP: Did you get an error like the following? failed to push some refs. You may miss some lfs files. Try to execute

    git lfs pull

  • Depending on the amount of data and the connection, the last step may require some time. When the command terminates, you can start working with your project on the target deployment.
    TIP: Be aware that user permissions (e.g. external collaborators or group permissions) aren’t preserved in the target deployment. You will need to add them manually. You will also need to activate the knowledge graph. You will see a warning the first time you open the cloned project. Just click on the “Activate Knowledge Graph” button.

Non-renku projects

Do you want to import projects from outside Renku? The steps are similar, but you may want to Renku-ize the project.
Our documentation can guide you through the necessary steps: https://renku.readthedocs.io/en/latest/tutorials/02_migrating_to_renku.html#renku-ize-your-project-with-renku-cli

4 Likes