Error when executing workflow


Regarding the following project Reproducible Data Science | Open Research | Renku

This post is connected to my previous post.

The project of the previous post and this project are forks of the same project. So the error could be connected. Although now the renku workflow visualize command still works but re running a workflow does not.

This project contains only one workflow and updating or executing it again gives the error below.
It seems like renku can not find the mentioned files but they are clearily still in the repository at their respective paths.

Is this a known error or do you have an idea what might cause it ?

  base ▶ ~ ▶ work ❯ niod-test ▶ master ▶ $ ▶ renku update --all
Resolved '../../../../tmp/tmp57eg4a00' to 'file:///tmp/tmp57eg4a00'
[job bbbb63ad-da40-4c68-8ad7-ca456f09a9fd] /tmp/xhblx2uq$ python3 \
    /tmp/xhblx2uq/ \
    it_tag_1 \
    2022-07-12 \
Traceback (most recent call last):
  File "/tmp/xhblx2uq/", line 77, in <module>
  File "/tmp/xhblx2uq/", line 26, in main
    collector = DataCollecter(testbed='m2c', 
  File "/home/jovyan/work/niod-test/modules/d01_utils/", line 17, in __init__
  File "/home/jovyan/work/niod-test/modules/d01_utils/", line 25, in read_configuration
    with open(self.configurationfile_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'configuration/sensor_configuration.yaml'
[job bbbb63ad-da40-4c68-8ad7-ca456f09a9fd] Max memory used: 118MiB
[job bbbb63ad-da40-4c68-8ad7-ca456f09a9fd] exited with status: 1
[job bbbb63ad-da40-4c68-8ad7-ca456f09a9fd] Job error:
("Error collecting output for parameter '_plans_1e17236b03e54ee1bde45b6d45259c31_outputs_82eb5c8c01b146ab88b896e55f498eb2': ../../../../tmp/tmp57eg4a00:37:5: Did not find output file with glob pattern: '['metrics_it_tag_1.csv']'.", {})
[job bbbb63ad-da40-4c68-8ad7-ca456f09a9fd] completed permanentFail
Error: Unable to finish executing workflow

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/renku/core/workflow/providers/", line 119, in workflow_execute
    outputs = process()
  File "/opt/conda/lib/python3.9/site-packages/cwltool/", line 34, in __call__
    raise WorkflowStatus(out, status)
cwltool.factory.WorkflowStatus: Completed permanentFail

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/renku/ui/cli/", line 92, in main
    return super().main(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/click/", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.9/site-packages/click/", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.9/site-packages/click/", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.9/site-packages/click/", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/renku/ui/cli/", line 195, in update
  File "/opt/conda/lib/python3.9/site-packages/renku/command/command_builder/", line 266, in execute
    hook(self, context, result, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/renku/command/command_builder/", line 202, in _post_hook
    raise result.error
  File "/opt/conda/lib/python3.9/site-packages/renku/command/command_builder/", line 252, in execute
    output = self._operation(*args, **kwargs)  # type: ignore
  File "/opt/conda/lib/python3.9/site-packages/renku/command/", line 84, in _update
    execute_workflow_graph(dag=graph.workflow_graph, provider=provider, config=config)
  File "/opt/conda/lib/python3.9/site-packages/inject/", line 342, in injection_wrapper
    return sync_func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/renku/core/workflow/", line 83, in execute_workflow_graph
    execute(dag=dag, basedir=project_context.path, provider=provider, config=config)
  File "/opt/conda/lib/python3.9/site-packages/renku/core/plugin/", line 95, in execute
    return executor(dag=dag, basedir=basedir, config=config)
  File "/opt/conda/lib/python3.9/site-packages/pluggy/", line 265, in __call__
    return self._hookexec(, self.get_hookimpls(), kwargs, firstresult)
  File "/opt/conda/lib/python3.9/site-packages/pluggy/", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/opt/conda/lib/python3.9/site-packages/pluggy/", line 60, in _multicall
    return outcome.get_result()
  File "/opt/conda/lib/python3.9/site-packages/pluggy/", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/opt/conda/lib/python3.9/site-packages/pluggy/", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/opt/conda/lib/python3.9/site-packages/renku/core/workflow/providers/", line 121, in workflow_execute
    raise WorkflowExecuteError() from e
renku.core.errors.WorkflowExecuteError: Unable to finish executing workflow

Workflows get re-executed in a temporary directory, copying all relevant files to there before execution. I’m guessing that the configuration/sensor_configuration.yaml file is used in code, but does not show up on the command line (so not detected by renku).

You can add these kinds of files as explicit inputs, as specified in the Specifying auxiliary inputs section of the docs.

Thanks a lot. This was exactly the problem and specifying the input solved it.