# Workflow recording bug: plans vs plans-by-name **URL:** \ **Category:** Renku (CLI)\ **Created:** [12 May 2022 09:36 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605 "2022-05-12T09:36:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jkminder](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jkminder/32/448_2.png) [@jkminder](https://renku.discourse.group/u/jkminder)\ **Post date:** [12 May 2022 09:36 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/1 "2022-05-12T09:36:28Z")
Hi, I’ve been using the `renku run` to run several scripts in parallel (on my local machine). Upon finishing all of them only part of the workflows are visible under `renku workflow ls`. All of the runs have finished successfully as I even got the print “Info: Adding these files to Git LFS:…”. If i run `renku log` i see all activities, also the runs that have not been recorded under `renku workflow ls`. If I manually check the renku metadata, the plans are all recorded in the `.renku/metadata/plans` file, but in the `.renku/metadata/plans-by-name` file only a subset is recorded (i guess this is what `renku workflow ls` prints). Have i done anything wrong? How can I sync these two files? You can download my .renku folder here: [polybox](https://polybox.ethz.ch/index.php/s/QBSs5YvV8pjq3Mk) Thanks for any help! ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [12 May 2022 09:40 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/2 "2022-05-12T09:40:22Z")
Hi there, can you give an example of some of the commands you are running? And how exactly do you run them in parallel? ---
**Author:** ![jkminder](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jkminder/32/448_2.png) [@jkminder](https://renku.discourse.group/u/jkminder)\ **Post date:** [12 May 2022 10:39 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/3 "2022-05-12T10:39:40Z")
Parallel means i just started the command multiple times in different terminals, so separate processes, with different parameters. Examples are `renku run --name parliamentdb_fetch_lp50 -- python conversion/database_fetch.py 50 -c -n` `renku run --name parliamentdb_fetch_lp49 -- python conversion/database_fetch.py 49 -c -n` The script basically just downloads data from a database to a local file, such that i can work offline on it later. ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [12 May 2022 11:35 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/4 "2022-05-12T11:35:07Z")
We don’t really support running renku commands in parallel at the moment. Running commands in parallel can cause one process to overwrite the results of another one, losing data in the process. We do have an `--isolation` flag to run a renku command in isolation (in a separately checked out git worktree), but merging back the metadata from the worktree to the repo will lead to merge conflicts. We have a custom git mergetool in the works that will be included in the next release that will support merging renku metadata. That said, the correct way to do what you want to do would be something like (you need to install the toil backend for renku to benefit from proper parallelization, by doing `pip install renku[toil]`) ```auto $ renku run --name parliamentdb_fetch -- python conversion/database_fetch.py 1 -c -n $ renku workflow show parliamentdb_fetch #check the output of this for the name of the parameter '1', I'll assume `param-2` $ renku workflow iterate -p toil --map param-2=[2,3,4,5,6,7,8,9,10,...,50] parliamentdb_fetch ``` That would execute your Plan `parliamentdb_fetch` 49 times, once for each value specified in interate. There’s no reason to create a new Plan for each execution usually, so this would work just fine. ---
**Author:** ![jkminder](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jkminder/32/448_2.png) [@jkminder](https://renku.discourse.group/u/jkminder)\ **Post date:** [12 May 2022 22:23 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/5 "2022-05-12T22:23:10Z")
Okay thanks a lot for your effort! Now another problem arises, if i execute the workflow without toil: `renku workflow iterate --map parameter-2=[39,40] parliamentdb_fetch` I get the following error: ```auto from lxml import etree ModuleNotFoundError: No module named 'lxml' ``` So somehow it must create a new environment that doesn’t contain some of the installed packages. If i run your commands (with toil) i get problems with local imports in the python script (I import python modules found in the same folder as the script itself under conversion/) → `ModuleNotFoundError: No module names....`. Running the script directly or via renku run this is no issue. ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [13 May 2022 07:12 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/6 "2022-05-13T07:12:55Z")
> [@jkminder](#): > > So somehow it must create a new environment that doesn’t contain some of the installed packages. It doesn’t create a new virtual environment and it does preserve the environment variables you are executing it with, so it should run the same as when you run it locally. What tool do you use to create/manage your virtual environment? > [@jkminder](#): > > If i run your commands (with toil) i get problems with local imports in the python script (I import python modules found in the same folder as the script itself under conversion/) → `ModuleNotFoundError: No module names....` . Running the script directly or via renku run this is no issue. This would probably also be an issue without toil. The way it works is that it copies all input files over to a temporary staging directory and runs the workflow there. Technically the other scripts are also inputs to the workflow, but since they don’t show up on the initial `renku run` shell command and they aren’t modified, they are not detected by renku to be relevant. You can specify them as auxiliary inputs as mentioned in [https://renku.readthedocs.io/en/latest/renku-python/docs/reference/commands.html#detecting-input-paths](https://renku.readthedocs.io/en/latest/renku-python/docs/reference/commands.html#detecting-input-paths) then they will be copied over before execution. If they’re in a subfolder, you can also just declare the folder as an input instead of having to specify each file, like `--input source=src/`. ---
**Author:** ![jkminder](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/jkminder/32/448_2.png) [@jkminder](https://renku.discourse.group/u/jkminder)\ **Post date:** [13 May 2022 12:50 UTC](https://renku.discourse.group/t/workflow-recording-bug-plans-vs-plans-by-name/605/7 "2022-05-13T12:50:19Z")
Thanks. So I resolved most of the stuff. I manage my environments with conda and pip but there was a version missmatch between renku and cwltool, so this works now.