Tracking .py file created in repository

I am currently working in a repository where I create a .py file with equations from a dataset, and tracking it to that point works.
I then import the equations from this .py file in another notebook, where I use it in combination with another dataset to create a graphical output. When looking at the workflow, the dataset input is there but the .py is not.
Is there a way to track importing internal .py files like this?

Hi @ Kriegelw,

You can define those files as explicit inputs to your command and renku will mark them as dependencies: renku run --input path/to/ .... See for more details.

It’s also possible to define those explicit inputs in your script using renku API: If you are not sure what approach to chose, then use command line argument for the moment since it does not require modifying your scripts.

Kind regards,

Hi @kriegelw,
You could use papermill, e.g.:

renku run papermill notebooks/notebook.ipynb \
notebooks/notebook.ran.ipynb -p importfile

whereas notebook.ipynb uses the following code to import

with warnings.catch_warnings():
    mod = importlib.import_module(basepath+importfile[:-3])
names = getattr(mod, '__all__', [n for n in dir(mod) if not n.startswith('_')])
g = globals()
for name in names:
    g[name] = getattr(mod, name)
1 Like