# Renku update: cwl error **URL:** \ **Category:** Renku (CLI)\ **Created:** [26 May 2021 13:39 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338 "2021-05-26T13:39:36Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [26 May 2021 13:39 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/1 "2021-05-26T13:39:36Z")
Hi, I’m working on a [test project](https://renkulab.io/projects/anthony.sonrel/sched-test-6) to create and latter update a dataset on Renku. To explain you a bit how the dataset is created: - The dummy dataset to be saved in the knowledge graph is created from the `src/dummy_data6.R`, which basically simulates a matrix and some metadata and save them in `data/dummy_data6/`. - `src/load_dataset` creates the renku dataset and run the `.R` script. If the data are already here, they are updated with `renku update`. The first run of `src/load_dataset.sh` executes fine and saves the dataset in the knowledge graph. However, if I change the metadata of one of the file from the `R` script, `renku update` fails, which seems to be related to `cwl` (see the error message [here](https://renkulab.io/gitlab/anthony.sonrel/sched-test-6/-/blob/master/log/load_dataset.out)). I’m a bit puzzled because I use the same scripts with other data, where only the content of the matrix and the metadata change, and the update runs fine. Do you know where the error related to `renku update` could come from and how I could resolve it ? Thanks in advance for you help! Anthony ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [26 May 2021 13:45 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/2 "2021-05-26T13:45:26Z")
Hi Anthony. The problem is that your script uses absolute paths, but `renku update` uses cwl to re-execute the workflow, which it does in a temporary directory. It expects outputs to be created in that temporary directory. I.e. it’s looking for something like `/tmp/tmpk6vcyr58/data/dummy_data6/meta_dummy.json` that it can copy back to the project directory, but instead the script created `/work/project/data/dummy_data6/meta_dummy.json`. So it complains that it couldn’t find an expected output in the temporary working directory. I hope that helps. Regards, Ralf ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [26 May 2021 15:46 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/3 "2021-05-26T15:46:29Z")
Thanks for your suggestion Ralf! So I’ve removed the dataset from the project (`renku dataset rm`), the data files and the absolute paths from all scripts. However I still have a problem with `cwl` (updated log [here](https://renkulab.io/gitlab/anthony.sonrel/sched-test-6/-/blob/master/log/load_dataset.out)). Do you maybe know if it is the same error, or if I need to clean the project in another way ? (maybe `git reset --hard`, but I’m not sure if it takes care of the data uploaded on the knowledge graph…) ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [26 May 2021 16:29 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/4 "2021-05-26T16:29:04Z")
Did you record the workflow again? I see a `define_workflow.sh` that only gets executed if `workflow_exists = False`. So it might be that the recorded workflow still refers to the old version of your script. ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [27 May 2021 14:47 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/5 "2021-05-27T14:47:21Z")
The error still happens unfortunately, although I’ve manually run the `define_workflow.sh` to create a new workflow. I’ve also created a ‘clean copy’ of the project to rerun everything from scratch (with relative paths), but the same error message appears. ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [27 May 2021 15:01 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/6 "2021-05-27T15:01:08Z")
Is there any reason why the command wouldn’t produce `data/dummy_data6/counts_dummy.mtx.gz` when run? Because it seems that file isn’t being created. I’m not too familiar with R, so I’m not sure I can be of help there, unfortunately. In the error message, if you check e.g. `("Error collecting output for parameter 'outputs_9d220fb7e262446ea0d9f7dcbe95ce4a':\n../../tmp/tmp97zty416/93fb75a0-1448-48da-b959-58b09adaf565.cwl:53:5: Did not find output file with glob pattern: '['data/dummy_data6/counts_dummy.mtx.gz']'", {}) ` then `/tmp/tmp97zty416/` should be the temporary directory it ran the workflow in. Though cwl is a bit weird, it uses multiple tmp directories (one for the script, one with the outputs I think) and I never remember which one shows up in the log. You could take a look inside that temp dir to see if you can find anything amiss. Maybe the file exists but is no in `data/dummy_data6/counts_dummy.mtx.gz` or maybe it wasn’t generated at all? Also keep in mind, `renku run` tries to detect inputs automatically and you can specify them manually as well (I saw you already use `--input` to declare a manual input). This is not just for the provenance, it is also to let CWL know which files it has to copy into the temp dir for the command to work. So if you script needs a file but this file doesn’t show up on the commandline (–\> so no automatic detection) and is not specified manually, your command might be failing on update because it’s missing an input? ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [27 May 2021 15:50 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/7 "2021-05-27T15:50:20Z")
Normally the `src/dummy_data6.R` script first creates a `data/dummy_data6/counts_dummy.mtx` which is then compressed to a `.gz` file (with an `overwrite=TRUE` option). I’ve tried it manually but apparently none of the intermediate or final files are empty. But maybe the presence of intermediate files can create the problem … ? I will have a deeper look at the temporary directory, and also at the `renku run` options. ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [10 June 2021 13:11 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/8 "2021-06-10T13:11:53Z")
Sorry for the late reply: So eventually my problem was indeed linked to a problem with the input. My main script (which was correctly flagged as input) was sourcing another script that was unknown to the `cwl` as it was not noticed by the initial `renku run`. I realize that the error message comes from `cwl` and that it was my mistake that I forgot to specify this other file as input, but I would maybe suggest to make `renku run` check if all required files are correctly listed as input ? For example in this case, it was a bit hard to debug the problem as `cwl` wrongly directed my research to a problem with the output where, in fact, it was due to a missing input. Anyway, thank you very much for your comments! I wouldn’t have been able to solve this issue without your last suggestion. ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [10 June 2021 13:21 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/9 "2021-06-10T13:21:07Z")
Glad to hear you could fix the issue! Renku doesn’t inspect scripts or anything of the sort, so all it sees is what was provided on the command line and what changed in the project directory after the command ran (detecting outputs). Since you could run arbitrary programs that do arbitrary things with renku, we really couldn’t detect all inputs that a script or program might use automatically. Even just with one programming language, this would be all but impossible, if it supports dynamic imports and the many ways there are of opening a file. Given that, I think the biggest issue is the very cryptic error message. Does the R command print an error message if that file is missing? Does it returns an exit code that suggest an error, i.e. something other than 0 (you can test this by executing the command and then running `echo $?` that should print the exit code)? Maybe we could show the command stdout/stderr when a rerun fails, to support users in debugging the issue. I’d have to look into cwl-runner to check if that’s possible. ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [10 June 2021 13:36 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/10 "2021-06-10T13:36:34Z")
Right, it’s true that it would not be feasible to detect all requirements/ inputs. I was more thinking about running a “fake” `renku update` after the initial `renku run`, to check whether all input are self-sufficient to reproduce the output ? (just random thought) So yes, I think that the error message is certainly the biggest issue. In my case, the R error message was quite straightforward: `cannot open file 'src/r_utils.R': No such file or directory.Execution halted` but the main challenge was to find this file somewhere in the cwl cache folders. I indeed think that a stdout/stderr would be very helpful 🙂 ---
**Author:** ![ralf.grubenmann](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ralf.grubenmann/32/78_2.png) [@ralf.grubenmann](https://renku.discourse.group/u/ralf.grubenmann)\ **Post date:** [10 June 2021 15:14 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/11 "2021-06-10T15:14:14Z")
Oh that’s a cool idea! Maybe with a flag like `--verify`. We’re working on adding a `--dryrun` option, this could go hand in hand with this. I’ll open an issue for this on our end. ---
**Author:** ![rrrrrok](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rrrrrok/32/117_2.png) [@rrrrrok](https://renku.discourse.group/u/rrrrrok)\ **Post date:** [10 June 2021 15:16 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/12 "2021-06-10T15:16:40Z")
can’t you use `renku rerun` for this? ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [11 June 2021 09:58 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/13 "2021-06-11T09:58:32Z")
At least on my script, it could be a solution, yes. While a run of `renku update` straight after my `renku run` (without any change to the input files) yields `All files were generated from the latest inputs.` running `renku rerun ` returns a bug. I could use this in my future scripts to check if all inputs are correctly set. ---
**Author:** ![rrrrrok](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rrrrrok/32/117_2.png) [@rrrrrok](https://renku.discourse.group/u/rrrrrok)\ **Post date:** [15 June 2021 16:22 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/14 "2021-06-15T16:22:35Z")
@ansonrel you mean it crashes because it can’t find one of the dependencies? Or it actually gives you a stacktrace? If it’s because of a missing input, it should give a better error than a stacktrace and it’s something we should fix. But yes, this was the original intent of `rerun`, i.e. to use it potentially as a part of continuous integration and verify that the project remains reproducible. ---
**Author:** ![ansonrel](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/ansonrel/32/243_2.png) [@ansonrel](https://renku.discourse.group/u/ansonrel)\ **Post date:** [21 June 2021 12:58 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/15 "2021-06-21T12:58:01Z")
Sorry, my knowledge in bash is limited and I’m not sure what you mean with “stacktrace” ? Basically, executing `renku rerun` without specifying a script that is sourced somewhere in the workflow will result in an error message finishing with: ```auto [job 0edc002c-a60a-4fd4-aed0-ad077b5a579f] Job error: ("Error collecting output for parameter 'outputs_9d220fb7e262446ea0d9f7dcbe95ce4a':\n../../tmp/tmp97zty416/93fb75a0-1448-48da-b959-58b09adaf565.cwl:53:5: Did not find output file with glob pattern: '['data/dummy_data6/counts_dummy.mtx.gz']'", {}) ERROR:cwltool:[job 0edc002c-a60a-4fd4-aed0-ad077b5a579f] Job error: ("Error collecting output for parameter 'outputs_9d220fb7e262446ea0d9f7dcbe95ce4a':\n../../tmp/tmp97zty416/93fb75a0-1448-48da-b959-58b09adaf565.cwl:53:5: Did not find output file with glob pattern: '['data/dummy_data6/counts_dummy.mtx.gz']'", {}) [job 0edc002c-a60a-4fd4-aed0-ad077b5a579f] completed permanentFail WARNING:cwltool:[job 0edc002c-a60a-4fd4-aed0-ad077b5a579f] completed permanentFail Error: Unable to finish re-executing workflow; check the workflow execution outline above and the generated /tmp/tmp97zty416/93fb75a0-1448-48da-b959-58b09adaf565.cwl file for potential issues, then remove the /tmp/tmp97zty416/93fb75a0-1448-48da-b959-58b09adaf565.cwl file and try again ``` which comes from this missing dependency (in this case, the script that is sourced somewhere in the workflow but wasn’t tagged as input). ---
**Author:** ![rrrrrok](https://yyz2.discourse-cdn.com/free1/user_avatar/renku.discourse.group/rrrrrok/32/117_2.png) [@rrrrrok](https://renku.discourse.group/u/rrrrrok)\ **Post date:** [21 June 2021 13:02 UTC](https://renku.discourse.group/t/renku-update-cwl-error/338/16 "2021-06-21T13:02:07Z")
I see, yes if it’s a missing dependency then it makes sense that you get an error on rerun. If you specify the script as a dependency with `--input` or similar, it should work. Incidentally, sounds like [this recent discussion](https://renku.discourse.group/t/renku-rerun-update-permission-error/349) was addressing the same problem 🙂