Manual definition of inputs and ouputs for lineage

1

Dear Renkuers,

Since it was implemented, I find that the most convenient way for me to create the lineage for a file is by defining in the scripts the inputs and outputs relative paths, and saving them into the files .renku/tmp/inputs.txt and .renku/tmp/outputs.txt, respectively. This has worked already in the past quite neatly.

However, I tried to do the same now, and even though all seems to work correctly, the files do not have any lineage. No idea which could be the reasons for that. Please, let me know who could take a look, and I will add you to the repo, as it is a private one.

Thank you so much
Best regards
Luis

2

it should be a yaml file like .renku/tmp/inputs.yml that looks like

my_file: file.csv
my_other_file: file2.csv

i.e. name:path pairs.

Out of curiosity, did you try the Renku API for this? It might be more convenient than using the files.

1 Like
3

Ohh, I see, so this changed! Cool then, I try with yaml, should be quick an easy. I guess no indent is required, each line is just an input, or an output, with the key-value pair, is that right? And regarding the API, I remember I tried in the past, but honestly, back then it did not work.

Thanks @ralf.grubenmann :raised_hands:

4

Yes, no indentation is needed and each line is for a file (or folder).

You could also use a workflow file which might be easier to use

1 Like
5

My reason to not use the Workflow files is that some of the input and output files have variable names that are defined inside the script, and using workflow files is not compatible with manual definition of inputs and outputs.

In any case, I gave it another try, using the API, and well, all went well (it seems…). But again, I got some output files that do not have any lineage. Hence, I don’t know what happened with all those Inputs, Outputs and Parameters defined.

The repo status presents a couple of errors, but it does not provide any further information.

image
image1059Ă—544 18.1 KB

Besides, there seems to be something weird with the datasets. The repo is not that old (less than 1 year), and in the beginning they were being shown correctly

image
image1282Ă—228 20.9 KB

Again, if at the end the problem is related with incompatible versions, it is a bit unfortunate, as the repo is not that old. How could I solve all these problems?

Thank you so much fellas for all your help!
Cheers

6

When you say “do not have any lineage”, do you mean in the UI? Do you see the lineage in the CLI using renku workflow visualize <path> ?

As to the UI issues, I guess you’re using renku CLI> 2.x on renkulab.io(not limited)? We still have to roll out Renku The Platform on renkulab.io so it properly supports Renku The CLI (should happen soon-ish) and until then, project using the newer version of the CLI have limited support

1 Like
7

Thanks for the reply again! This renku workflow visualize <path> worked! So yep, now I know at least the information is stored :wink:

Then, waiting for the new fancy roll out in order to visualize the graph in all its glory on the UI

Thank you so much!

1 Like