Datasets created but not seen

1

Hi,

I created a new project and added 2 datasets , but even if I can see the second dataset in the “datasets” tab it’s not displayed in the “File > data” tab and it’s not accessible when I start a session.

The file is a bit large, so it’s maybe just a matter of time for it to be loaded…

Thank you for your help

Romain

1 Like
2

Something very strange is going on with this project. I started it (thanks for the link!) and ran the following command:

$ renku dataset ls
ID    NAME    TITLE    VERSION
----  ------  -------  ---------

I would have expected to see the two datasets that are visible in the UI in the output of this command.

Let me loop in someone with more detailed knowledge of the component responsible to managing datasets and see what they say.

1 Like
3

The template you used for this project is invalid.

It contains renku-templates/metadata.yml at main · BIOP/renku-templates · GitHub and renku-templates/template_checksums.json at main · BIOP/renku-templates · GitHub files, which a template should not have (these are created by renku when creating a project).

Since it has a metadata.yml file for version 0.16.2, (which we don’t use anymore), it is both a renku-python 0.16.2 (because of metadata.yml) and a renku-python 1.2.2 (because of .renku/metadata/project · master · Romain Guiet / ABBA-BIOP_VM · GitLab that was created when the project was created) project at the same time, which isn’t valid. If you use renku-python <1.0.0, it’ll see the metadata.yml, think this is a valid 0.16.2 project, and check .renku/datasets/ folder for datasets (which doesn’t exists, so there’s no datasets in this project). If you use renku-python 1.2.2, it’ll correctly check .renku/metadata/datasets instead and list the datasets just fine:

$ renku --version
1.2.3
$ renku dataset ls
ID                                NAME                                                               TITLE                                                              VERSION
--------------------------------  -----------------------------------------------------------------  -----------------------------------------------------------------  ---------
5e71211c69e249f8b3925da4c36d5529  serial-coronal-sections-of-an-adult-sections-brain-sample-dataset  Serial Coronal Sections Of An Adult Sections Brain Sample Dataset
ddcae20bdf084189a3ce5aae09aefa7d  allen-mouse-ccf-v3-labels-modulo-65000                             Allen Mouse CCF v3 Labels Modulo 65000

Replacing the contents of .renku/metadata.yml with

# Dummy file kept for backwards compatibility, does not contain actual version
'http://schema.org/schemaVersion': '9'

and committing that should fix the issue.

Also, the .renku/metadata.yaml and .renku/template_checksums.json should be removed from renku-templates/BIOP-virtualmachine/.renku at main · BIOP/renku-templates · GitHub to make the template valid.

1 Like
4

Hi @ralf.grubenmann ,

thank you for your suggestions !

I was not sure what you meant with

so in a first commit I just replaced the corresponding line which didn’t do much… and in a second commit I erased everything which didn’t do much either…

So I modified the template as you suggested :

and I created a new project, I managed to add the “light” dataset with the import function , but if I try to do the same with the “heavy” dataset I get the error below :

image
image696×515 23.7 KB

Thank you for your help, it’s much appreciated !

Cheers

Romain

5

Yes, the second is what I meant. Are you working in interactive sessions or locally? You have to update the Renku CLI for this to work. If you do renku --version it should tell you the version. You can upgrade it locally by running pip install --upgrade renku or for the interactive sessions by modifying the Dockerfile here (Set to 1.2.4 or 1.2.2, both should be OK).

As for the template, this line also has to be changed to read ARG RENKU_VERSION={{ __renku_version__ | default("1.2.2") }}. That way, renkulab will automatically fill in the correct version when creating a project.

6

changing dockerfile didn’t do much… data · master · Romain Guiet / ABBA-BIOP_VM · GitLab

I modified the template and created a new project and I tried to use renku cli.

renku dataset import https://renkulab.io/datasets/ddcae20bdf084189a3ce5aae09aefa7d
works for the “light” dataset

but renku dataset import https://renkulab.io/datasets/5e71211c69e249f8b3925da4c36d5529
or renku dataset import https://doi.org/10.5281/zenodo.5018718
cause bugs Issues · SwissDataScienceCenter/renku-python · GitHub

Best,

Romain

7

Just to make sure you didn’t miss it, I replied on the issues you created.

1 Like