I ran now a couple of renku run commands, and get this rather complicated knowledge graph:
I noticed a couple of arrows pointing the wrong way. Or boxes without any errors. Now I wanted to check if everything worked fine with the inputs and outputs. Is there a way to quickly find the workflow for a certain file?
You could run renku workflow visualize <output_file> to check, that should show you what happened and allow you to check if all the dependencies are correct. With the -i flag you can move through the activities and press enter to get details, which includes the Plan (workflow) name.
We currently have this PR open feat(cli): show plans in renku log by mohammad-sdsc · Pull Request #2779 · SwissDataScienceCenter/renku-python · GitHub which would allow you to do something like renku log -w |grep -C 20 <filename> |grep -A 2 "Plan:" to get the Plan for a file.
The visualization code on renkulab is due for an overhaul, I’m not sure how correct it still is.
Okay, thank you, that helped! I found now that there seems to be something wrong with cyclic dependencies when I do renku workflow visualize data/FLEX_results/aus_prec0_2/AdelaideRiver_DS_DalyRoad/Eidt, I get the error:
Traceback (most recent call last):
File "/home/rnijzink/.local/bin/renku", line 8, in <module>
sys.exit(cli())
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/cli/exception_handler.py", line 128, in main
result = super().main(*args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/cli/exception_handler.py", line 92, in main
return super().main(*args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/cli/workflow.py", line 1171, in visualize
visualize_graph_command()
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/core/management/command_builder/command.py", line 242, in execute
output = context["click_context"].invoke(self._operation, *args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/rnijzink/.local/lib/python3.8/site-packages/inject/__init__.py", line 342, in injection_wrapper
return sync_func(*args, **kwargs)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/core/commands/workflow.py", line 574, in _visualize_graph
graph = create_activity_graph(activities, with_inputs_outputs=show_files)
File "/home/rnijzink/.local/pipx/venvs/renku/lib/python3.8/site-packages/renku/core/management/workflow/activity.py", line 209, in create_activity_graph
raise ValueError("Cannot find execution order: Project has cyclic dependencies.")
ValueError: Cannot find execution order: Project has cyclic dependencies.
But I am sure there are no cyclic dependencies. I inspected the workflow, and the inputs and outputs are fine. Any idea what might cause this?
I see
Activity /activities/f2d42bfa0d544ba682c6d47d7716cea2
Start Time: 2022-03-23T16:35:28+01:00
End Time: 2022-03-23T16:35:29+01:00
User: […]
Renku Version: renku 1.1.1
Command: python3 src_py/prep_flex_nml.py --inobjectives data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Obj_cal.txt --inparamfile data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Parameters.txt -ic data/aus_catch_chars/AdelaideRiver_DS_DalyRoad_catch_chars.txt -id data/HYDRO_input/aus_prec0_2/AdelaideRiver_DS_DalyRoad/Date.txt -oc work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/config.nml -op work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/param.nml
Inputs:
input-1: src_py/prep_flex_nml.py
inobjectives-2: data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Obj_cal.txt
inparamfile-3: data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Parameters.txt
input-5: data/aus_catch_chars/AdelaideRiver_DS_DalyRoad_catch_chars.txt
input-7: data/HYDRO_input/aus_prec0_2/AdelaideRiver_DS_DalyRoad/Date.txt
input-11: work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/param.nml
Outputs:
output-9: work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/config.nml
Parameters:
i-4: c
i-6: d
o-8: c
o-10: p
Activity /activities/9574b6e09e2f47aa9c9a69d6fe3e8057
Start Time: 2022-03-23T16:00:58+01:00
End Time: 2022-03-23T16:00:59+01:00
User: […]
Renku Version: renku 1.1.1
Command: python3 src_py/prep_flex_nml.py --it 50000 --inobjectives data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Obj_cal.txt --inparamfile data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Parameters.txt -ic data/aus_catch_chars/AdelaideRiver_DS_DalyRoad_catch_chars.txt -id data/HYDRO_input/aus_prec0_2/AdelaideRiver_DS_DalyRoad/Date.txt -oc work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/config.nml -op work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/param.nml
Inputs:
input-1: src_py/prep_flex_nml.py
inobjectives-3: data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Obj_cal.txt
inparamfile-4: data/FLEX_results/aus_prec1_0/AdelaideRiver_DS_DalyRoad/Parameters.txt
input-6: data/aus_catch_chars/AdelaideRiver_DS_DalyRoad_catch_chars.txt
input-8: data/HYDRO_input/aus_prec0_2/AdelaideRiver_DS_DalyRoad/Date.txt
input-10: work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/config.nml
Outputs:
output-12: work/FLEX/aus_prec0_2/AdelaideRiver_DS_DalyRoad/param.nml
Parameters:
it-2: 50000
i-5: c
i-7: d
o-9: c
o-11: p
emphasis mine. That produces a direct cycle.
Might be that an output was erroneously detected as an input? If the file existed before the execution and wasn’t modified, that would happen.
The commits this happened in are 803bdf997f4efbac38f7d6b4f720e4544e68886a and e7f96077ce196d8103f829c738a071aa6c2b5efc.
As for a fix, deleting those commits might fix things (Do it on a branch first, to see if everything works) and you could rerecord the commands and explicitly specify if something is an input or output with --input or --output specifically.
I’ve also created add cycle detection in `renku run` · Issue #2783 · SwissDataScienceCenter/renku-python · GitHub as you should not be able to execute a run that creates a cycle. And the check for that is clearly still there (hence renku workflow visualize failing), but somehow it didn’t raise an exception when this workflow was created.
Oka, thank you!. I did a hard reset to just before those two commits and re-did a couple of things. However, not I have a different error:
Traceback (most recent call last):
File "[...]/renku/cli/exception_handler.py", line 128, in main
result = super().main(*args, **kwargs)
File "[...]/renku/cli/exception_handler.py", line 92, in main
return super().main(*args, **kwargs)
File "[...]/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "[...]/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "[...]/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "[...]/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "[...]/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "[...]/renku/cli/workflow.py", line 1175, in visualize
text_output, navigation_data = result.output.text_representation(columns=columns, color=not no_color, ascii=ascii)
File "[...]/renku/core/commands/view_model/activity_graph.py", line 214, in text_representation
local_max_y, node_color = self._add_edges_to_canvas(
File "[...]/renku/core/commands/view_model/activity_graph.py", line 149, in _add_edges_to_canvas
if len(colors) < len(EdgeShape.EDGE_COLORS):
AttributeError: type object 'EdgeShape' has no attribute 'EDGE_COLORS'
Thank for letting me know.
I’ve prepared a hotfix that will prevent renku run, renku workflow execute and renku workflow iterate from creating cycles in workflows and as the bug you mentioned above is an easy fix, I’ve included that as well. It’s currently on chore: release 1.1.4 by Panaetius · Pull Request #2785 · SwissDataScienceCenter/renku-python · GitHub waiting for a review, but you could test that branch already and visualize should work as expected.
It’s been release, so you can upgrade to renku · PyPI
This version fixes the bug you had above and raises an error if a renku run would result in a circular workflow graph.
Ah super, thank you!