Renku-python v1.0.1: Output path recognized as output instead of output file

Hi all,

this is slightly connected to this post, but a more general problem.

I’m running a workflow, where I don’t know the explicit output name when calling renku run using renku-python version 1.0.1. Usually renku does automatically detect all new files and I don’t need to worry about explicitly specifying the output. In this case I parse the path to the output file as command line option. Renku seems to detect the output path as output instead of the actual output file. This gives rise to problems when executing the workflow with a different set of parameter etc. and will also not work if files exist in the output directory.

In this project I have a test case (workflow param_5):
The test script (src/parameter_outpath2.sh) has two input arguments -o, which is the path to the output file and -p with some parameter to write to to the output file’s content.
When I run it using:

renku run --name param_5 -- bash src/parameter_outpath2.sh -o "data/test2" -p "test outpath"

a workflow is generated, but I get this note:

and renku workflow show looks like:

with the output directory defined as output instead of the output file itself.
Interestingly the same happens when running renku run with --no-output-detection and explicitly specifying the output file:

Is there a way to circumvent this problem and call renku run while parsing the output path to the script to run, e.g., by being recognized as parameter instead of output?

Best,
Almut

Hi @Almut,

Thanks for reaching to us with this issue!

Renku tries to guess inputs and outputs based on the parameters that are passed in the command line. Since data/test2 is passed, renku uses it as an output instead of the file inside it.

At the moment, it’s not possible to mark a command line argument as a workflow parameter. I’ll create a story for your use-case to either allow defining parameters or not requiring an empty directory when doing run/rerun.

Meanwhile, you can try to disable both input and output detection and pass them as explicits:

renku run --name param_7 --no-input-detection --no-output-detection --input src/parameter_outpath2.sh --output data/test3/test_out2.txt -- bash src/parameter_outpath2.sh -o data/test3 -p test outpath

This generates the following workflow:

$ renku workflow show param_7
Id: /plans/88b67dc129ea445bb38eb7b797b79767
Name: param_7
Command: bash src/parameter_outpath2.sh -p test outpath
Success Codes: 
Inputs: 
        - input-1:
                Default Value: src/parameter_outpath2.sh
                Position: 1
Outputs: 
        - output-1a33:
                Default Value: data/test3/test_out2.txt
                Position: None
Parameters: 
        - p-3:
                Default Value: test
                Position: 3
                Prefix: -p 
        - parameter-4:
                Default Value: outpath
                Position: 4

Kind regard,
Mohammad

1 Like