No lineage information

1

Dear Renku team,

I ran some simulations, but it keeps saying “No lineage information”. It is about these files:

https://renkulab.io/projects/wave/tuw-natt-catchments-budyko/files/lineage/data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/Objectives.txt

Do you know what went wrong? The workflow seems okay:

Success Codes: 
Inputs: 
	- input-1:
		Default Value: src_r/run_tuw_aus.r
		Position: 1
	- input-2:
		Default Value: data/HYDRO_input/AdelaideRiver_DS_DalyRoad/prec1_0/Prec.txt
		Position: 2
	- input-3:
		Default Value: data/HYDRO_input/AdelaideRiver_DS_DalyRoad/prec1_0/Etp.txt
		Position: 3
	- input-4:
		Default Value: data/HYDRO_input/AdelaideRiver_DS_DalyRoad/prec1_0/Temp.txt
		Position: 4
	- input-5:
		Default Value: data/HYDRO_input/AdelaideRiver_DS_DalyRoad/prec1_0/Qobs.txt
		Position: 5
	- input-6:
		Default Value: data/HYDRO_input/AdelaideRiver_DS_DalyRoad/prec1_0/Date.txt
		Position: 6
Outputs: 
	- output-10:
		Default Value: data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/parbest.txt
		Position: 10
	- output-8:
		Default Value: data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/Qbest.txt
		Position: 8
	- output-9:
		Default Value: data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/Ebest.txt
		Position: 9
	- output-12:
		Default Value: data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/Params.txt
		Position: 12
	- output-11:
		Default Value: data/TUW_results/AdelaideRiver_DS_DalyRoad/prec1_0/Objectives.txt
		Position: 11
Parameters: 
	- parameter-7:
		Default Value: 50000
		Position: 7
2

Hi @rcnijzink,

So it looks like we couldn’t process successfully any commit after migration you did on 2022-02-25. That’s why there’s almost no data for your project in the Knowledge Graph. Let us try to find out the reason for that.

3

Could it have to do with it that it is an Rscript running in the python-template of renku? Also the re-run commands fail as it doesn’t find the R-packages, so I thought those issues might be linked?

4

From the error on our end it looks like there was an issue migrating the project, but we’d need access to the project to further determine what went wrong.

5

Yes, something was strange there. I first migrated locally, but it kept saying it was outdated on renkulab.io. So then I migrated there too and pulled it to my local repo.

6

Could be a merge conflict when you pulled to your local repo.
We store metadata files in .renku/metadata/ inside the project, which is made up of many files and index files pointing to those files.
The error we see is something along the lines of Cannot find object: 'dc636d5349fc40dc65c7448eae16115b877baab04cbc5035b4474f284f82c69b' which means we have a metadata file somewhere in .renku/metadata/ pointing to a file named dc636d5349fc40dc65c7448eae16115b877baab04cbc5035b4474f284f82c69b, but that .renku/metadata/dc/63/dc636d5349fc40dc65c7448eae16115b877baab04cbc5035b4474f284f82c69b doesn’t exist (i.e. the renku metadata is corrupt).
Git doing a merge of an index file pointing to dc63[…] but not merging the actual file could explain that.
But impossible to tell without looking at the git history in question.

7

Guess this is where it went wrong then:

commit cb153482fcbaa231886d563bb5c823cc465aa5ba
Merge: e6bdbb1 33cd078
Author: Remko NIJZINK <remko.nijzink@list.lu>
Date:   Fri Feb 25 16:58:23 2022 +0100

    XMerge branch 'master' of renkulab.io:wave/tuw-natt-catchments-budyko

commit 33cd078dc9a37a51fa3831018d4b39306d357ed8
Author: Remko Nijzink <remko.nijzink@list.lu>
Date:   Fri Feb 25 15:58:10 2022 +0000

    renkulab.io: migrate to latest version for renku 1.0.6
    
    renku-transaction: 6111f01a79604e87b02a70f76167ce6f

commit e6bdbb1ccd06b38a74765bb2e23f0c2b5c7a54cf
Author: Remko NIJZINK <remko.nijzink@list.lu>
Date:   Fri Feb 25 16:57:35 2022 +0100

    renku migrate
    
    renku-transaction: ec82843626024f5dae644716b5892b3b

Would it be a solution to roll-back to before the migration, and then migrate again?

8

Yes that should work, if the migration/merge was the culprit.

Or git reset --hard <commit> to the migration commit (i think 33cd078dc9a37a51fa3831018d4b39306d357ed8 in this case) from renkulab (discarding the local merge).

9

Okay will give that a try. I also gave you access now, as I have actually the same issue here:

https://renkulab.io/projects/wave/gr4j-natt-catchments-budyko/files/lineage/data/GR4J_results/AdelaideRiver_DS_DalyRoad/prec1_0/Objectives.txt

And there I had no issues with the migration.

I also found that some data that I imported, are also not in the knowledge graph, maybe there is something wrong there? It is this data:

https://renkulab.io/projects/wave/gr4j-natt-catchments-budyko/datasets/HYDRO_input/

with the original data here:

https://renkulab.io/projects/wave/natt-catchments-hydro-inputs/datasets/HYDRO_input/

10

Thank you for giving me access. This is due to a bug that was fixed in fix(dataset): unset wasDerivedFrom for imported datasets by mohammad-sdsc · Pull Request #2686 · SwissDataScienceCenter/renku-python · GitHub (but not in any release yet) that we thought didn’t affect users, but apparently it does. Namely it breaks renku graph export (used for getting the Knowledge Graph) and renku log in your case.
We’re working on adding a renku doctor fix that can fix affected projects and on making the affected parts of the code more robust.
This manifests when importing a dataset with renku-python version 1.0.6.
We’ll let you know once the renku doctor fix is available.

11

We made a release 1.1.0 last Friday, renku · PyPI, you can use that to call renku doctor --fix to fix an affected project.

The project should still work with 1.0.6 currently deployed on Renkulab after that, no migration is needed.

But any datasets newly imported from Renkulab will still have this issue until we make an overall release of the Renku platform. Until then, users have to call renku doctor --fix with 1.1.0 after importing a dataset from Renkulab and being affected by this issue.

12

Thank you! I tried it now and it fixed it indeed :slight_smile: