# Track origins of files in datasets?

**URL:** <https://renku.discourse.group/t/track-origins-of-files-in-datasets/283>\
**Category:** General\
**Created:** [21 March 2021 13:47 UTC](https://renku.discourse.group/t/track-origins-of-files-in-datasets/283 "2021-03-21T13:47:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\
**Post date:** [21 March 2021 13:47 UTC](https://renku.discourse.group/t/track-origins-of-files-in-datasets/283/1 "2021-03-21T13:47:57Z")

</div>

I had to download a file (HowardSprings\_2020\_L3.nc) manually as I could not get a direct link to it, and I placed it in the folder of the respective dataset, then I tried to add it by `renku dataset add HowardSprings temp/HowardSprings_2020_L3.nc`. This failed with a warning that I should provide a URL for proper tracking. So I contacted the data supplier and he gave me a direct link to a complete file, `HowardSprings_L3.nc`. I ran `renku dataset add HowardSprings URL`, forgetting that I still had the untracked `HowardSprings_2020_L3.nc` in the folder. What happened, is that apparently renku added and committed that file first, with the commit message `renku dataset: committing 1 newly added files` and then downloaded the second file, `HowardSprings_L3.nc` and commited with the commit message `renku dataset add HowardSprings http://dap.ozflux.org.au/thredds/fileServer/ozflux/sites/HowardSp...`. Wouldn’t it be better to warn the user that the repo is dirty before taking executive decisions?

This is all in the repo `https://renkulab.io/projects/ba_math_gen-16/et-data`. When I browse the datasets, I find both files, but there is no lineage information for either of them. When I look at `HowardSprings_L3.nc` in the gitlab interface, it points to the commit `renku dataset: committing 1 newly added files` instead of the `renku dataset add` commit. But then, the latter does not contain the path anyway, so I am totally lost at how to recover the origin of the files in the dataset. Could anyone help me out here? I was trusting that the lineage gets recorded when running `renku dataset add`, since this was implicit in the error message of my first attempt, but if this is not the case, I will have to include it for each file in a readme. Not sure how `renku dataset update` should work if the link is truncated, though.