Hacker News new | past | comments | ask | show | jobs | submit login

Sqlite has an archive format that is worthwhile to be aware of

https://sqlite.org/sqlar.html




It unfortunately has major limitations on file size for use cases where that matters. Max file size is on the order of 2^32 bytes since files are stored in a single SQLite blob. Note that this limit is only for input file size. The archive file can basically be of any arbitrary size.

Personally I wish sqlar didn't have this limitation since SQLite is awesome for tooling.


It seems pretty easy to build big files variant of sqlar; instead of a single sqlar table, have index and filedata tables; the index has all the elements of the normal sqlar table except data, and adds a counter for the number of chunks the file is stored in (a directory is zero chunks, an empty file is zero-length with one, empty, chunk), the filedata has composite PK consisting of an FK to the index and an integer sequence number in the file, and a data blob. The code changes to store/extract from it should be straightforward, and given the size of SQLite integers it would in theory structurally handle file sizes (as stored, which may be compressed) of up to 2^63-1 bytes; but the SQLite 281 TB database size limit would be reached well before this.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: