Workflow recording bug: plans vs plans-by-name

Renku (CLI)
1

Hi,
I’ve been using the renku run to run several scripts in parallel (on my local machine). Upon finishing all of them only part of the workflows are visible under renku workflow ls. All of the runs have finished successfully as I even got the print “Info: Adding these files to Git LFS:…”. If i run renku log i see all activities, also the runs that have not been recorded under renku workflow ls. If I manually check the renku metadata, the plans are all recorded in the .renku/metadata/plans file, but in the .renku/metadata/plans-by-name file only a subset is recorded (i guess this is what renku workflow ls prints). Have i done anything wrong? How can I sync these two files? You can download my .renku folder here: polybox

Thanks for any help!

1 Like
2

Hi there,

can you give an example of some of the commands you are running?
And how exactly do you run them in parallel?

3

Parallel means i just started the command multiple times in different terminals, so separate processes, with different parameters. Examples are
renku run --name parliamentdb_fetch_lp50 -- python conversion/database_fetch.py 50 -c -n
renku run --name parliamentdb_fetch_lp49 -- python conversion/database_fetch.py 49 -c -n
The script basically just downloads data from a database to a local file, such that i can work offline on it later.

4

We don’t really support running renku commands in parallel at the moment. Running commands in parallel can cause one process to overwrite the results of another one, losing data in the process.

We do have an --isolation flag to run a renku command in isolation (in a separately checked out git worktree), but merging back the metadata from the worktree to the repo will lead to merge conflicts. We have a custom git mergetool in the works that will be included in the next release that will support merging renku metadata.

That said, the correct way to do what you want to do would be something like (you need to install the toil backend for renku to benefit from proper parallelization, by doing pip install renku[toil])

$ renku run --name parliamentdb_fetch -- python conversion/database_fetch.py 1 -c -n
$ renku workflow show parliamentdb_fetch  #check the output of this for the name of the parameter '1', I'll assume `param-2`
$ renku workflow iterate -p toil --map param-2=[2,3,4,5,6,7,8,9,10,...,50] parliamentdb_fetch

That would execute your Plan parliamentdb_fetch 49 times, once for each value specified in interate. There’s no reason to create a new Plan for each execution usually, so this would work just fine.

1 Like
5

Okay thanks a lot for your effort! Now another problem arises, if i execute the workflow without toil:
renku workflow iterate --map parameter-2=[39,40] parliamentdb_fetch I get the following error:

    from lxml import etree
ModuleNotFoundError: No module named 'lxml'

So somehow it must create a new environment that doesn’t contain some of the installed packages.

If i run your commands (with toil) i get problems with local imports in the python script (I import python modules found in the same folder as the script itself under conversion/) → ModuleNotFoundError: No module names..... Running the script directly or via renku run this is no issue.

6

It doesn’t create a new virtual environment and it does preserve the environment variables you are executing it with, so it should run the same as when you run it locally. What tool do you use to create/manage your virtual environment?

This would probably also be an issue without toil. The way it works is that it copies all input files over to a temporary staging directory and runs the workflow there. Technically the other scripts are also inputs to the workflow, but since they don’t show up on the initial renku run shell command and they aren’t modified, they are not detected by renku to be relevant. You can specify them as auxiliary inputs as mentioned in https://renku.readthedocs.io/en/latest/renku-python/docs/reference/commands.html#detecting-input-paths then they will be copied over before execution. If they’re in a subfolder, you can also just declare the folder as an input instead of having to specify each file, like --input source=src/.

1 Like
7

Thanks. So I resolved most of the stuff. I manage my environments with conda and pip but there was a version missmatch between renku and cwltool, so this works now.

2 Likes