# How to find the source of an imported dataset?

**URL:** <https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956>\
**Category:** Renku (CLI)\
**Created:** [22 November 2023 17:31 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956 "2023-11-22T17:31:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [22 November 2023 17:31 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/1 "2023-11-22T17:31:11Z")

</div>

Could someone tell me how to find out what was the source of an imported dataset?  
I imported a dataset from github (ESSM\_plotting) and then tried `renku dataset show ESSM_plotting`, but this only shows:

```auto
Name: ESSM_plotting
Created: 2023-11-20 19:03:35+01:00
Data Directory: modules/imported/ESSM_plotting
Creator(s): Stan Schymanski <stan.schymanski@env.ethz.ch>
Keywords: 
Version: 
Storage: 
Annotations: 
Title: ESSM_plotting
Description: 

```

How do I get the original github link?

---

<div class="post-metadata">

**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\
**Post date:** [23 November 2023 09:20 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/2 "2023-11-23T09:20:00Z")

</div>

I don’t think there’s an easy way to see the. Your best bet is probably to use `renku dataset ls --format json-ld`, find the entry for the dataset and check its `sameAs` property.

---

<div class="post-metadata">

**Author:** ![jachro](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jachro/32/161_2.png) [@jachro](https://renku.discourse.group/u/jachro)\
**Post date:** [23 November 2023 10:09 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/3 "2023-11-23T10:09:17Z")

</div>

Hi @schymans.  
I’m guessing you are using our renku CLI. However, if it would be an option for you to resort to our API, you can call `https://renkulab.io/knowledge-graph/datasets/63154e15a55b4db1abbf4ea868e7111c` and in the response, you can find out what was the previous version of the dataset (the `derivedFrom`) or the initial one (the `versions.initial` or even follow the link to `initial-version` listed under the `_links` property).

---

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [27 November 2023 08:20 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/4 "2023-11-27T08:20:05Z")

</div>

Hi @jachro, Thanks for your help. Unless I made a mistake somewhere, the dataset should be linked to [GitHub - schymans/ESSM\_plotting: Python modules to simplify plotting of equations generated using ESSM](https://github.com/schymans/ESSM_plotting), but I don’t see any mention of this link in the metadata. Perhaps I deleted the wrong one when trying to re-import and removing the old, local dataset?

---

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [27 November 2023 08:25 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/5 "2023-11-27T08:25:31Z")

</div>

> [@ralf.grubenmann](#):
>
> renku dataset ls --format json-ld

Thanks @ralf.grubenmann, I do find the link in the output, but I would never be able to find it if I didn’t know what to look for. Considering that transparent re-use of data and giving credit to the sources of datasets is an important motivation for renku, it would be very important to see these links easily and prominently. Is this on the todo-list for the next renku update?

---

<div class="post-metadata">

**Author:** ![laura.kinkead](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/laura.kinkead/32/402_2.png) [@laura.kinkead](https://renku.discourse.group/u/laura.kinkead)\
**Post date:** [27 November 2023 09:40 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/6 "2023-11-27T09:40:52Z")

</div>

Hi @schymans, Thank you for your feedback! I agree that making dataset provenance easily understandable is important for Renku, and not something we do a good job at currently! This is not on our immediate roadmap (which you can find [here](https://github.com/SwissDataScienceCenter/renku-design-docs/blob/main/roadmap.md), by the way), but we are planning to overhaul Renku’s dataset concept in 2024.

---

<div class="post-metadata">

**Author:** ![jachro](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jachro/32/161_2.png) [@jachro](https://renku.discourse.group/u/jachro)\
**Post date:** [27 November 2023 11:29 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/7 "2023-11-27T11:29:56Z")

</div>

Hi @schymans. Actually, the way the command you suggest in your GitHub repo works might be a bit different to what you could be expecting. When you initialise a Dataset this way

```auto
renku --no-external-storage dataset add --create ESSM_plotting \
git@github.com:schymans/ESSM_plotting.git

```

a new Dataset is created and files from the provided GitHub repo are copied. Although info about the origins of each file is added to the metadata (on the level of each file), the Dataset is still perceived as a created (not imported) Dataset, and no link saying that’s the same as the GitHub Dataset exists. We’d only add such a link (on the Dataset level) when the `dataset import` command is used. I could guess the `import` command could be more suitable in your case but unfortunately, currently, it’s not possible to issue such a command with a link to a git repository.

---

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [1 December 2023 12:49 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/8 "2023-12-01T12:49:48Z")

</div>

Aaah, thank you! So the clean way would be to publish the dataset on zenodo first and then import from zenodo? So could I e.g. execute:  
`renku --no-external-storage dataset import --datadir modules/imported/ https://zenodo.org/records/4380979`?

---

<div class="post-metadata">

**Author:** ![jachro](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jachro/32/161_2.png) [@jachro](https://renku.discourse.group/u/jachro)\
**Post date:** [1 December 2023 16:53 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/9 "2023-12-01T16:53:32Z")

</div>

> [@schymans](#):
>
> renku --no-external-storage dataset import --datadir modules/imported/ [https://zenodo.org/records/4380979](https://zenodo.org/records/4380979)

Yes, this command should do the job but, unfortunately, recent changes on the Zenodo API cause our import to fail. I know @ralf.grubenmann was looking into this but cannot say what’s the status now.

---

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [22 January 2024 19:50 UTC](https://renku.discourse.group/t/how-to-find-the-source-of-an-imported-dataset/956/10 "2024-01-22T19:50:52Z")

</div>

Hmmm, I just tried it out again using renku 2.7.0, and `renku dataset add` does actually preserve the link. Here is what I did:

````auto
renku --no-external-storage dataset add modules --source modules/* git@renkulab.io:wave/li-6800.git```

````

Today, I found a bug, so I modified a file in the modules folder in [renkulab.io](http://renkulab.io):wave/li-6800. Then I ran:

```auto
(main *%)stan:~/notebooks/jupyter/renku/fortress$ renku dataset update modules
Checking git files for updates: 0%| | 0.00/1.00 [00:00<?, ?iB/s]Cloning remote repository...
Updated 1 files 

```

The changed file is now updated in the repo where I executed these commands, so obviously the link is preserved even if I do `renku dataset add` instead of `renku dataset clone`. The command `renku dataset ls --format json-ld`, as suggested by @ralf.grubenmann also worked very well, showing the links to the origins of the different files in the datasets. Thanks!
