Plus a build script running in a GitHub actions workflow that compiles the notes into a SQLite file using my markdown-to-sqlite tool and publishes the resulting SQLite file using Datasette to https://til.simonwillison.net - which gives me search and an Atom feed and suchlike.
The site has custom templates so it Durant look like regular Datasette, but you can run custom queries against it at https://til.simonwillison.net/tils
this is very interesting.
Would you still use this architecture if u were building something from scratch today ?
I can see the markdown format being powerful. But how do you parse it and create a document graph ? (or rather...what do u use to persist the document graph)
I have been trying to use Firebase on a side project of mine for this markdown -> graph problem
I don’t see the difficulty here — with SQLite, you’d just store the set of links as an adjacency list; for the major job of linking things, you don’t even need to traverse the graph — you only need to know 1-level of relationships to generate the links. But when you do traverse, you can use recursive WITH.
Presumably for each build, you’d parse the names of each doc to produce the list of nodes (and their paths), and then as you encounter the markdown reference, update it accordingly.
If you start afresh each time, handling updates would be trivial, but you’re at risk of destroying existing URLs when changing doc titles/reorganizing — so you probably want a canonical name (to generate the final URL for), and the symbolic names (to refer by in markdown).
And then eventually you realize you want multiple ways to organize your docs, so you start tagging, and then eventually you realize you want ways to reference groups of notes, so you’ll go ahead and implement hierarchal tagging, and you’ll finally have be able to treat your graph as a graph, instead of a forest (list of trees)
And finally you can simplify the whole thing to only reference tags (or parent tags), which may happen to link to a single doc, or multiple. Probably in the special case of 1 doc, the link directly takes you to the doc instead of some collection page.
A document store seems like a terrible idea, because you’re really not modeling a tree, and that’s probably the majority of your modeling problem — I imagine you’re trying to currently stuff a graph into a tree.
My TILs site runs uses a GitHub repository where the notes live in markdown: https://github.com/simonw/til
Plus a build script running in a GitHub actions workflow that compiles the notes into a SQLite file using my markdown-to-sqlite tool and publishes the resulting SQLite file using Datasette to https://til.simonwillison.net - which gives me search and an Atom feed and suchlike.
The site has custom templates so it Durant look like regular Datasette, but you can run custom queries against it at https://til.simonwillison.net/tils