Hacker News new | past | comments | ask | show | jobs | submit login

I had the same trouble with order dependence as notebooks got to a certain size, so my team and I created and open-sourced a library, Loman, to help with that. It allows you to interactively create a graph, where nodes represent inputs or functions, and then keeps track of state as you change or add inputs, intermediate functions and request recalculations. Our experience has been broadly positive with this way of working. As graphs get larger, it's easy to lift them into code files in libraries, while continuing to modify or extend them in notebooks. The graph structure and visualization make it easy to return to loman graphs with up to low hundreds of nodes, which would make for a fearsome notebook otherwise. It also makes it easy to bolt Qt or Bokeh UIs onto them for interactive dashboards - just bind UI widgets and events to the inputs and widgets to the outputs. They can be serialized, which is useful for tracking exceptions in intermediate calculations when we put them in airflow to run periodically, as you can see all the inputs to the failing calculation, and its upstreams.

[1] Github: https://github.com/janushendersonassetallocation/loman [2] Quickstart/Docs: https://loman.readthedocs.io/en/latest/user/quickstart.html




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: