# Add code from different project **URL:** \ **Category:** General\ **Created:** [16 April 2020 16:54 UTC](https://renku.discourse.group/t/add-code-from-different-project/89 "2020-04-16T16:54:22Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [16 April 2020 16:54 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/1 "2020-04-16T16:54:22Z")
I would like to be able to transparently import code files from other projects, similar to `renku dataset add`. For example, I would like to use [https://renkulab.io/projects/remko.nijzink/vomcases/files/blob/src\_py/plot\_et\_ass.py](https://renkulab.io/projects/remko.nijzink/vomcases/files/blob/src_py/plot_et_ass.py) in a different project, but be clear about where I took it from. Should i just pretend that plot\_et\_ass.py is a dataset and use `renku dataset add`, or is there a more appropriate function to do this? I was also thinking of submodules, but this would require that plot\_et\_ass.py would be in a separate git repo. ---
**Author:** ![cramakri](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/cramakri/32/7_2.png) [@cramakri](https://renku.discourse.group/u/cramakri)\ **Post date:** [17 April 2020 15:32 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/2 "2020-04-17T15:32:17Z")
Hi! As you point out, using a submodule would be one solution. I think this is probably the best solution if you want to use the entire other project, but if you just want a file or two, it could be overkill. The semantics of renku dataset add [file from a git repo] are pretty close to what you want here, but the main problem is that it would put the file into git-lfs by default, and it is cumbersome (though not impossible) to store the file directly in git. I will see if anyone here has a better idea and get them to contribute here. ---
**Author:** ![mohammad-sdsc](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/mohammad-sdsc/32/79_2.png) [@mohammad-sdsc](https://renku.discourse.group/u/mohammad-sdsc)\ **Post date:** [17 April 2020 17:52 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/3 "2020-04-17T17:52:03Z")
Hello, ATM, what @cramakri proposed is the best solution. We have an open issue to allow adding/sharing/tracking code similar to what can be done with data: [https://github.com/SwissDataScienceCenter/renku-python/issues/671](https://github.com/SwissDataScienceCenter/renku-python/issues/671). ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [18 April 2020 09:52 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/4 "2020-04-18T09:52:44Z")
Thanks a lot, @cramakri and @mohammad-sdsc! I had not seen the issue. Just added a comment to it, as it seems to be concerning data files only, not code. As @cramakri pointed out, placing a code file in git-lfs is counter-productive. Would it not be easy to copy the functionality of `renku dataset add` but remove the git-lfs part? Or add an option to `renku dataset add` to let the user choose where the file is placed, and maybe allow a tag for code? ---
**Author:** ![rrrrrok](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rrrrrok/32/117_2.png) [@rrrrrok](https://renku.discourse.group/u/rrrrrok)\ **Post date:** [20 April 2020 12:30 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/5 "2020-04-20T12:30:41Z")
> [@cramakri](#): > > the main problem is that it would put the file into git-lfs by default, and it is cumbersome (though not impossible) to store the file directly in git. This has been fixed in [`v0.10.0`](https://github.com/SwissDataScienceCenter/renku-python/releases/tag/v0.10.0) - you need to include the `--no-external-storage` flag to renku commands like this ```bash renku --no-external-storage dataset add ... ``` ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [17 July 2020 17:13 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/6 "2020-07-17T17:13:14Z")
Thanks so much for the update on this. It worked like a charm, except that I just got tripped up by an error message that git lfs was not installed. Could it be added to the instructions for installing renku, so that the users know that they need to install git-lfs separately? ---
**Author:** ![rrrrrok](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rrrrrok/32/117_2.png) [@rrrrrok](https://renku.discourse.group/u/rrrrrok)\ **Post date:** [19 August 2020 22:49 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/7 "2020-08-19T22:49:53Z")
Hi @schymans, sorry for taking a while to respond and thanks for the suggestion - I’ve made an issue to add this to the docs: [https://github.com/SwissDataScienceCenter/renku/issues/1396](https://github.com/SwissDataScienceCenter/renku/issues/1396) ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [16 February 2021 21:50 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/8 "2021-02-16T21:50:13Z")
I am still struggling a little with the following workflow: I take a jupyter notebook from another repo and then modify it to work with my new repo, but I want to record the original provenance of the file for giving proper credit at the end. I can do this with `renku --no-external-storage dataset add ...` as suggested earlier, but this will put the file into `data/externaldataset1/...`, even if I use the `--destination` option, whereas I keep my jupyter files in `jupyter/...`. What would be the best way of moving or copying the file to my jupyter folder transparently? ---
**Author:** ![rcnijzink](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rcnijzink/32/136_2.png) [@rcnijzink](https://renku.discourse.group/u/rcnijzink)\ **Post date:** [12 November 2021 12:26 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/9 "2021-11-12T12:26:53Z")
I am having a bit of similar issues with sharing code across projects. Now I wondered if it would be helpful to collect scripts, that I use more often, in a python package. In that case, I could just add the package and version in requirements.txt and update this if I make changes to some scripts. But will renku also notice that outputs are generated with an older version of a certain package? ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [21 November 2021 16:02 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/10 "2021-11-21T16:02:26Z")
Good idea, but to include it in requirements.txt, you would need to put the package on pypi, right? And then I don’t think that renku would track which version of a pypi package was used for a given result. How about `renku --no-external-storage dataset add ...` followed by `renku mv` to put the script where you want it? ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [3 December 2021 20:40 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/11 "2021-12-03T20:40:26Z")
HOORAY! This actually works, I just tried it out. I created some .py scripts in `git@renkulab.io:wave/li-6800.git` in the folder `modules_exported`, then imported them in another one as: ```auto renku --no-external-storage dataset add modules --source modules/* git@renkulab.io:wave/li-6800.git ``` Then I moved them from `data/modules` to `modules` in the receiving repo: ```auto renku mv data/modules/ modules/ ``` Now I updated one of the .py files in the original repo, commited, then executed `renku dataset update modules` in the receiving repo and the file was updated there and in the right place! Awesome! ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [15 April 2023 22:14 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/12 "2023-04-15T22:14:10Z")
Unfortunately, this approach does not work any more: ```auto $ renku --no-external-storage dataset add --create modules --source modules/* git@renkulab.io:wave/li-6800.git Error: Invalid parameter value - Cannot use '--source' with multiple URLs. ``` Is there another way now, or do I have to list every single file in the folder? ---
**Author:** ![mohammad-sdsc](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/mohammad-sdsc/32/79_2.png) [@mohammad-sdsc](https://renku.discourse.group/u/mohammad-sdsc)\ **Post date:** [17 April 2023 08:05 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/13 "2023-04-17T08:05:14Z")
Hi @schymans, The reason you get this error is that you have a `modules` directory in the root of your project and your shell expands the wildcard. To avoid the shell from expanding, just put `modules/*` inside single quotes (double quotes won’t work): ```auto renku --no-external-storage dataset add --create modules --source 'modules/*' git@renkulab.io:wave/li-6800.git ``` Let me know if this doesn’t fix your issue. ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [21 April 2023 15:10 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/14 "2023-04-21T15:10:15Z")
Thank you, @mohammad-sdsc , this works! I probably would have never found this simple solution. I’ve been running a python script to add every file individually… ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [20 November 2023 12:17 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/15 "2023-11-20T12:17:02Z")
In renku 2.7.0 the above procedure does not seem to work any more: ```auto $ renku --no-external-storage dataset add --create ESSM_plotting git@github.com:schymans/ESSM_plotting.git Error: Cannot find file Cannot find '/home/stan/notebooks/jupyter/WAVE/renkulab/LI-6800/.renku/cache/github.com/schymans/ESSM_plotting.git/.git/config' in the remote project ``` The file renku complains about definitely exists: ```auto $ more /home/stan/notebooks/jupyter/WAVE/renkulab/LI-6800/.renku/cache/github.com/schymans/ESSM_plotting.git/.git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [submodule] active = . [remote "origin"] url = git@github.com:schymans/ESSM_plotting.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "main"] remote = origin merge = refs/heads/main [lfs] repositoryformatversion = 0 [filter "lfs"] clean = git-lfs clean -- %f smudge = git-lfs smudge --skip -- %f process = git-lfs filter-process --skip required = true ``` @mohammad-sdsc @cramakri do see what I might have overlooked? Or should I use a different method now? ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [20 November 2023 12:44 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/16 "2023-11-20T12:44:49Z")
OK, the original command still works, but not without `--source 'modules/*'`. This also works: ```auto renku dataset add --create ESSM_plotting --source '*' git@github.com:schymans/ESSM_plotting.git ``` Why not without `--source`? I thought that option is only needed if we want to limit ourselves to a subfolder. Is this a bug? ---
**Author:** ![mohammad-sdsc](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/mohammad-sdsc/32/79_2.png) [@mohammad-sdsc](https://renku.discourse.group/u/mohammad-sdsc)\ **Post date:** [20 November 2023 18:46 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/17 "2023-11-20T18:46:53Z")
Hello @schymans You must always specify files (i.e. sources) that will be added from a git repository. This means that you cannot call `renku dataset add` without using the `--source` option (when adding data from a git repository). There’s no default value to add the whole git repository if no source is specified. I’ll create an enhancement issue to allow this in future. ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [14 December 2023 15:16 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/18 "2023-12-14T15:16:06Z")
Argh, there is another problem. Previously, I could just execute `renku mv data/modules/ modules/` to move the entire dataset, but if I execute this command now, I get a warning that the files will be removed from the dataset. So the location of the dataset is now hard-wired? ---
**Author:** ![schymans](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/schymans/32/77_2.png) [@schymans](https://renku.discourse.group/u/schymans)\ **Post date:** [21 May 2024 12:23 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/19 "2024-05-21T12:23:04Z")
Just ran into this problem again. Is there a way to move code imported as ‘dataset’ out of the ‘data’ folder without removing it from the dataset itself? Otherwise, it gets confusing to have code in the folder ‘data’. I used to be able to use `renku mv` on a dataset, but now I get: `Warning: You are trying to move dataset files out of a datasets data directory. These files will be removed from the source dataset: ...` ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [21 May 2024 12:29 UTC](https://renku.discourse.group/t/add-code-from-different-project/89/20 "2024-05-21T12:29:53Z")
you can set a dataset’s data directory when you create it, see [renku dataset — Renku documentation](https://renku.readthedocs.io/en/stable/renku-python/docs/reference/commands/dataset.html#cmdoption-renku-dataset-create-datadir) . You can’t change it once a dataset is created, but you can create a new dataset with a data dir you want and move files into it (if they’re already in a dataset, otherwise just add them) using renku mv with this flag: [renku mv — Renku documentation](https://renku.readthedocs.io/en/stable/renku-python/docs/reference/commands/move.html#cmdoption-renku-mv-to-dataset) [Next page](https://renku.discourse.group/t/add-code-from-different-project/89.md?page=2)