Hi all,
I’m testing the new renku workflow features in renku-python version 1.0.1. In general it is really great and much easier for me to interact with a projects workflows.
One thing that I came across when trying to execute a workflow with a different set of parameter is that I always need to explicitly specify the new expected output, if the output name depends on the parameter settings. For example this workflow will generate a file named by the input parameter h-2 and w-3:
We try to write some functions to execute a specific workflow with different sets of parameter independent of how the output file name depends on the parameter settings.
So I was wondering if it was possible to have something like the automatic output detection of renku run for renku workflow execute as well?
Or maybe is there a way to bypass this problem that I’m not aware of, e.g., by using placeholder or similar?
based on your renku workflow show param_3 for the output-e321 does not have a position, which is because the output is actually not explicitly specified on the command line, rather it’s detected once the workflow has been executed and detected that there’s a new file - as you have mentioned.
I’m gonna look into this locally. it would help me if you could share the repository that contains the above example, but of course I can mimic such a workflow myself.
Note currently what you are doing, i.e. having the input parameters as template variables for the output file name, e.g data/file_param_{h-2}_{w-3}.txt is actually in plan for implementing: renku workflow design documentation, but hasn’t gotten into 1.0.0 release.
The repository for this workflow is this one.
It works, when using --set output-321=... as expected. The problem is that we ultimately want the script to be provided by a user, while we provide the code to run and execute the workflows. That is why it will be hard to automatize execution across parameter, while defining the output explicitly. Still I that feature would help a lot.
For another test here, that is a bit more similar to what we actually plan to run I tried to write the output to .renku/tmp/output.txt within the script that was executed by the workflow, but that didn’t work either.
Let me know, if you need any more infos or if I should try something else.
Best,
Almut
i mean what you want is an automatic output detection without having the output explicitly specified. This is not that straightforward, and currently i dont see a clear path to resolve this, unless you would be ok with a template variable based output specification, that i’ve mentioned earlier.
could you please elaborate this part, as i’m not so sure if i understand what exactly was your aim to put the output under .renku/tmp
For another test here, that is a bit more similar to what we actually plan to run I tried to write the output to .renku/tmp/output.txt within the script that was executed by the workflow, but that didn’t work either.
and could you please try to provide examples that are as basic as possible? the repo you’ve provided above has at least 2 shell scripts and various R scripts.
Hi Viktor,
thanks for your reply. I see that this is a more complex problem than I thought, because output detection of renku run works so smooth.
Sorry I probably expressed this wrong - I think I can find a way to work with the template variable based output specification, if automatic detection is not possible.
Anyways here is a simple example project for this - just in case.
Re .renku/tmp/output.txt: I didn’t write the output to .renku/tmp but specified the output name in a file called .renk/tmp/output.txt within the script run by the workflow as described in the renku documentation under “Specifying inputs and outputs programmatically”. Did I misunderstand it?
