I forked an old project (from 2021), successfully ran renku migrate, but now if I run renku status, it shows a lot of outdated files, but I cannot update them, as:
Hi @schymans, As far as I understand, the way the workflows were defined, Renku may be erroneously detecting an input as an output. Basically, I see that at least two files (theory_equation.py and theory_variable.py) are inputs and outputs of you workflows (run renku workflow inputs and renku workflow outputs for checking input and output files). I assume that they should not be part of the outputs, since scripts are usually input files. Please, specify the wrongly detected outputs as an explicit input using āāinputā when defining the workflows.
Thanks, @elisabetc! In fact both files are outputs and inputs, but this should not be a circularity. Essentially the files are created out of notebooks where new mathematical formulations are derived, then written to .py files and imported elsewhere to be applied to data. renku workflow inputs and renku workflow outputs just lists the files without displaying any directional interdependencies, so this does not help detect circularity, unfortunately.
@elisabetc do you have any other ideas how I could resolve this alleged circularity problem? I assume that the circularity checking was introduced more recently, and that it just blindly tests whether any files are listed as outputs and inputs anywhere, without checking for the sequence of actions in the workflow. Could this be a bug?
Hi @schymans, I will check the different steps in the workflow, since it should not happen that the circularity error appears if the outputs of a step are the inputs for a following step. The circularity works checking inputs and outputs of the same step or consecutive steps (e.g. the input of a former step is the output of a latter one). I will look again and let you know.
what this error means is that activity 615a8439d53c8d899f6ae07c6660df421988c52d has notebooks/theory/pyFile_storage/theory_variable.py as output, which is an input of activity bf074d031839a74fce6b2f93ec92fe6e4ba9f2d8 which has notebooks/theory/pyFile_storage/theory_equation.py as an output which in turn is an input to 615a8439d53c8d899f6ae07c6660df421988c52d again, forming a circle.
You could do renku workflow revert /activities/615a8439d53c8d899f6ae07c6660df421988c52d or renku workflow revert /activities/bf074d031839a74fce6b2f93ec92fe6e4ba9f2d8 to undo (delete) either of those activities and that should fix the issue. You can call them with --metadata-only to not undo file changes, just the recorded provenance. You probably also need --force as you can otherwise remove activities that have dependents.
It could be worthwhile to check what those activities are beforehand to better understand the problem. We unfortunately donāt have a command to show the details of an activity by id, but renku log -w shows details of all activities and you can grep for the idās to see what the activities were, including the command.
(sorry for the formatting)
The papermill command passes notebooks/theory/pyFile_storage/theory_variable.py and output_path_equation notebooks/theory/pyFile_storage/theory_equation.py as parameters, but notebooks/theory/pyFile_storage/theory_equation.py was recognised as input file. Do you have an idea how this could have happened? The renku run command should have modified both files, so I donāt understand how one could have been seen as input.
I think if the file was there before the renku run and the content didnāt change it recognizes it as an input. You can also force it to be an output with the --output <path> parameter , like renku run --output myfile -- python script.py myfile
Thanks, I didnāt realise that folders provided after -p would also be recorded as input folders. So if, instead of running renku update, I run the renku run command again, and the output files passed on the command line donāt change, they will be falsely recorded as input files?
Thatās a bummer, as I also made that mistake several times. If I run renku workflow ls and it shows several workflows with the same command, I should probably remove the duplicates, right?
I just checked again, and there are lots of duplicate workflows. Would it not make sense to check each renku run command before execution and warn the user if it is a duplicate of an existing workflow? Or even make renku status part of renku run and return a warning if anything is out of date? Then it would become clear if a recently modified file (e.g. script.py) is already associated with a workflow that does what the user wants to do.
Yes, you need to revert at least for one activity in each cycle to break it (It looks like some activities are in multiple cycles so it could be less than 20).
And yes, you are correct, I wanted to write can't there.
Awesome, thanks, will try. Would it be worth a feature request to check for duplicate activities when running renku run? Or at least to recommend a renku status before contemplating a renku run?
We already have a check in renku run that would fail if there is a cycle, Iām not sure why this didnāt catch this early. It uses the same code as the check in renku workflow visualize so itās surprising that it passed in run and then fails in visualize.
Was the workflow in question created before you migrated the project? It could be that we didnāt check this when the workflow was initially run and that the migration also doesnāt check for cycles, that could explain it.
The current Renku CLI is mostly in maintenance mode at the moment, as we are working on a new CLI for the new Renku platform. For the new CLI we plan to have easier and more robust detection of provenance by intercepting syscalls, which would allow us to better detect inputs and outputs. Though we probably wonāt have our own workflow language, as we hope that the new CLI could gather provenance with any workflow tool a user wants to use.
But so if thereās a critical bug, weāre happy to fix it but new feature requests probably wont happen unless theyāre critical.
Thanks a lot, @ralf.grubenmann! Indeed, these alleged cycles were created many years ago, probably before any of this checking was in place. I will try to clean it up as suggested abve.
Sorry, another question, before I get started on reverting. The activities listed as cycles have different hashes than the workflows returned by renku workflow ls. What is the difference between a workflow and an activity, and do I have to remove both the activity and the associated workflow that created each cycle? As I mentioned above, there are multiple workflows doing the same thing, which likely caused the problem in the first place, so I am not entirely clear yet how to clean this up without the problem re-occurring later.
An activity is a historical record of a single execution of a workflow. A workflow could be run multiple times and it would create a new activity each time. Deleting a workflow only soft-deletes it and keeps the activities around, but for cycles, only the activities matter, so only activities are what needs to be deleted, not the workflow itself.
I think keeping the workflows around isnāt an issue, as rerunning them would fail cycle detection.
That does sound like a bug, Iāll have to look into that. It looks like it doesnāt delete it from all places that renku keeps track of. Thank you for the bug report.
It might get fixed by running renku doctor --fix which should reindex all activities properly, as a workaround.
By the way, you probably also want the --metadata-only flag when deleting the activities as otherwise it also tries to undo changes made to files by the activity (like git revert ... essentially).
Thanks! Unfortunately, renku doctor --fix does not make a difference. It only gives a warning about large files and recommends to run renku storage migrate. After this, renku status returns the same error as before.
ā¦