# Datasets Disappearing **URL:** \ **Category:** RenkuLab\ **Created:** [7 April 2022 14:09 UTC](https://renku.discourse.group/t/datasets-disappearing/586 "2022-04-07T14:09:37Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![PaulineML](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/paulineml/32/476_2.png) [@PaulineML](https://renku.discourse.group/u/PaulineML)\ **Post date:** [7 April 2022 14:09 UTC](https://renku.discourse.group/t/datasets-disappearing/586/1 "2022-04-07T14:09:37Z")
Hi, We are working on a project which had 4 Datasets (originally uploaded through the UI with ‘Add Dataset’). Sometimes our person of contact was adding files to the UI with the ‘modify dataset’ button. However, lately he was not able to add any data anymore as there were no datasets visible anymore in the UI in the Datasets tab. It was showing a ‘No dataset found for this project’. Fortunately, this was only in the UI and all the files are still available in the Renku session. I created a new dataset, which is ‘visible’ and he is now able to add files to this new dataset through the UI again. Would it be possible to fix this and make the 4 datasets visible from the interface again ? There are 2 potential reasons for this (but maybe it is not due to any of these): - We do not like to have the raw data in the data/ folder directly. So every time raw files were added, we moved them to another folder in data/raw/ and deleted the automatically created new folder in data. – For example, for the dataset ‘vbv\_data’, when files were added, in the renku session a folder named ‘vbv\_data’ was created in data/vbv\_data. We moved these files to data/raw/vbv\_data and deleted the now empty folder data/vbv\_data. **By the way, would it be possible to let an option for this in Renku by choosing a sub folder in which to store each datasets like data/raw/ instead of data/ automatically ?** - Our default branch is develop, not master. So when there are updates on Renku, we need to manually merge them in develop. We could have missed one or done something in the wrong order. Thank you for your help, Pauline ML ---
**Author:** ![cramakri](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/cramakri/32/7_2.png) [@cramakri](https://renku.discourse.group/u/cramakri)\ **Post date:** [8 April 2022 07:06 UTC](https://renku.discourse.group/t/datasets-disappearing/586/2 "2022-04-08T07:06:47Z")
Thank you for throughly describing the situation: your information is very helpful for debugging the issue. I have some ideas about what could be the cause of this problem, but it would be helpful to look at the project to verify that. Would it be possible to give me, `cramakri`, access to this project and send me a link to it (by DM if you prefer)? If that is not feasible, that is also OK – I think I can recreate the scenario that is causing problems – but if it is possible, it would be a little faster. Thanks! ---
**Author:** ![PaulineML](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/paulineml/32/476_2.png) [@PaulineML](https://renku.discourse.group/u/PaulineML)\ **Post date:** [19 April 2022 10:34 UTC](https://renku.discourse.group/t/datasets-disappearing/586/3 "2022-04-19T10:34:32Z")
Thanks for the fixes. The datasets are now visible again. ---
**Author:** ![cramakri](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/cramakri/32/7_2.png) [@cramakri](https://renku.discourse.group/u/cramakri)\ **Post date:** [19 April 2022 12:38 UTC](https://renku.discourse.group/t/datasets-disappearing/586/4 "2022-04-19T12:38:44Z")
I’m glad that could get sorted out! 🎉 ---
**Author:** ![lusamino](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/lusamino/32/227_2.png) [@lusamino](https://renku.discourse.group/u/lusamino)\ **Post date:** [13 May 2022 12:24 UTC](https://renku.discourse.group/t/datasets-disappearing/586/5 "2022-05-13T12:24:31Z")
Hi to all, Would it be possible to know how this problem was solved? I have just created a new project [https://renkulab.io/projects/luis.salamanca/democrasci-parsedpdfs](https://renkulab.io/projects/luis.salamanca/democrasci-parsedpdfs) And even though locally, through the CLI, I have created 4 datasets seamlessly, and added data to them, those dataset do not appear on the UI. I preferred to follow-up here as these errors might be related, but in any case I can see many errors related to datasets recently, so perhaps I should have gone somewhere else. Thank you so much! ---
**Author:** ![cramakri](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/cramakri/32/7_2.png) [@cramakri](https://renku.discourse.group/u/cramakri)\ **Post date:** [13 May 2022 12:31 UTC](https://renku.discourse.group/t/datasets-disappearing/586/6 "2022-05-13T12:31:13Z")
Hi Luis! Given the age of your project, which is quite new, I am pretty sure that the situation that you are seeing is different from the one that PaulineML had. I think the problem you are having is probably the same as [Dataset is not in KG](https://renku.discourse.group/t/dataset-is-not-in-kg/607). That problem is still being investigated, but you should see the follow-up in the other post (and we can notify you as well). ---
**Author:** ![lusamino](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/lusamino/32/227_2.png) [@lusamino](https://renku.discourse.group/u/lusamino)\ **Post date:** [13 May 2022 12:35 UTC](https://renku.discourse.group/t/datasets-disappearing/586/7 "2022-05-13T12:35:16Z")
Awesome! I will keep an eye on that thread then! In any case, I can see locally the datasets, and all the metadata associated, so I assume things are working correctly. I know they will just pop up magically on the UI 🙂 On the meantime, I stay vigilant. Thank you so much! ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [13 May 2022 13:34 UTC](https://renku.discourse.group/t/datasets-disappearing/586/8 "2022-05-13T13:34:51Z")
@lusamino The project you created through the UI was created using `renku-python` 1.2.2. But then locally you have 0.16.0 installed and you added datasets with that. But 0.16.0 stores datasets in a different way that 1.x.x cannot read. So if you install 1.2.4 locally and use that to create the datasets, everything should work (also, delete the `.renku/datasets/` folder). You should not have been able to run 0.16.x on a project created with 1.x.x. but it looks like the code that prevents this was only added when creating a project on the command line (`renku init`), not when creating a project through the service. ---
**Author:** ![lusamino](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/lusamino/32/227_2.png) [@lusamino](https://renku.discourse.group/u/lusamino)\ **Post date:** [16 May 2022 07:27 UTC](https://renku.discourse.group/t/datasets-disappearing/586/9 "2022-05-16T07:27:04Z")
Thanks @ralf.grubenmann for the answer! Actually, this is another super confusing thing for me. Let’s hope I can explain myself. I remember a long time ago I installed Renku with the `pipx` commands provided here [GitHub - SwissDataScienceCenter/renku-python: A Python library for the Renku collaborative data science platform.](https://github.com/SwissDataScienceCenter/renku-python) Then, if I do `which renku`, I can perfectly locate the path, and all seems to work. However, before starting this new repo, I checked the version and it was on `0.14.0`. Therefore, I decided to upgrade, and the command I found was `pipx upgrade renku`, and that took me to `0.16.0`. Today I run it again and it prompts this ```auto $ pipx upgrade renku renku is already at latest version 0.16.0 (location: /Users//.local/pipx/venvs/renku) ``` But, for the new repository, I created a new Python environment (using `conda env`). In that new environment I cannot see `renku` through `pip`. So I decided to install it in the new env using standard `pip install renku==1.2.2`. And in this case, I can force the version up to 1.2.2, something that I could not with `pipx`. Therefore, now I am extremely confused. Should I use `pipx` for installing `Renku` as you suggest? But that will pin me to version `0.16.0`. Or should I go with `pip` in each repo, that allow me going up to 1.2.4.? Sorry for the weird questions, but honestly, this is super confusing to me. Thank you so much! Cheers Luis ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [16 May 2022 08:03 UTC](https://renku.discourse.group/t/datasets-disappearing/586/10 "2022-05-16T08:03:25Z")
We have switched our images from using `pipx` to using `pip`. `pipx` has internal references to `wheel` and `setuptools`, among others, which it just automatically updates every 30 days, and there’s no way to pin those. So `pipx` will just update those at random times for different users, and we had issues in the past where some version of `renku-python` didn’t work with a new version of `setuptools` and `pipx` would just upgrade to that with no way for us to prevent that, breaking users environments. Hence we now use `pip` in a separate python environment in our Dockerfiles, so we can properly pin all version of all packages. But you can continue using `pipx` in your local environments, if you want. With some of that background info out of the way, the most likely reason why you can’t `pipx upgrade renku` is that your `pipx` uses your system Python version and that is Python 3.6, and we dropped support for Python 3.6 in `renku-python` version 0.16.1. If you upgraded to Python 3.7, `pipx` should happily upgrade `renku-python` to 1.2.2. Inside the conda environment, conda probably installed a more current version of Python, hence you could install 1.2.x. Please note that Python 3.6 reached it’s end of life on 23.12.2021, so it’s highly encouraged that you upgrade to at least 3.7, which has an EOL of 27.06.2023. ---
**Author:** ![lusamino](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/lusamino/32/227_2.png) [@lusamino](https://renku.discourse.group/u/lusamino)\ **Post date:** [16 May 2022 08:06 UTC](https://renku.discourse.group/t/datasets-disappearing/586/11 "2022-05-16T08:06:28Z")
Thanks @ralf.grubenmann !! Then, that fully clarifies everything, and now I can perfectly tackle the issue. Thank you so much again! Btw, I guess that, if I just remove `.renku/datasets`, I can start again the creation without any further issues, is that right? (I have not done any `renku run` yet) Cheers Luis ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [16 May 2022 08:13 UTC](https://renku.discourse.group/t/datasets-disappearing/586/12 "2022-05-16T08:13:15Z")
> [@lusamino](#): > > Btw, I guess that, if I just remove `.renku/datasets` , I can start again the creation without any further issues, is that right? (I have not done any `renku run` yet) Removing that folder is more of a cleanup thing. Renku \>= 1.0.0 just complete disregards it, so there’s no reason to keep it around. So it’d work even without removing it. What I definitely would do is add a file `.renku/metadata.yml` with this content: ```auto # Dummy file kept for backwards compatibility, does not contain actual version 'http://schema.org/schemaVersion': '9' ``` This should have been created automatically when you created the project, but there was a bug(fixed but not released yet) that meant it only worked when doing `renku init` to create a project, not when creating a project through the [renkulab.io](http://renkulab.io) UI. This is only there for backwards compatibility and what it does is make renku-python \<1.0.0 refuse to work on the project (which would have prevented you running into the original problem 🙂 ). So just remove `.renku/datasets/` and add this file in a commit and all should be golden.