Hacker News new | past | comments | ask | show | jobs | submit login

Regarding database hosting, if you would consider giving the data away, I would suggest converting it to an SQLite database and sharing it over Torrent.



I'm guessing OP wants to share a database that's always up to date.

A torrent containing a single sqlite file would be good for a snapshot in time, but each update would require a new torrent, even if it only contains the updates since the base or last release.

IIRC IPFS can be used to distribute files that change over time, with only the changes being transferred, although of course there would need to be a place where OP publishes the hash of the most recent file.

In either case, someone would need to seed the file to guarantee it's always available.


I second this. You've done a great service to collect this data. I'm guessing the file must be much smaller than 20GB when compressed.


I've also did an experiment by generating and searching embeddings for all the comments on HN. Here is the walkthrough: https://www.youtube.com/watch?v=hGRNcftpqAk


It is only around 5 GB in ClickHouse. Details: https://github.com/ClickHouse/ClickHouse/issues/29693




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: