Good question, in the current version, compaction is left to the caller.
The library does a few things to keep key growth practical:
1. generate_n_keys_between: if you know you're inserting N items at once (e.g. a bulk paste or drag of multiple items), it distributes them evenly in a single call rather than chaining N sequential generate_key_between calls, which would produce progressively longer keys.
2. Higher bases: base62 (default) and base95 give you a much wider branching factor per character than base10, so keys stay shorter under the same insertion patterns.
But yes, in a long-lived list with adversarial insertion patterns (always inserting at the same spot), keys will grow unboundedly, that's inherent to the algorithm. In practice, compaction is an application-level concern: you'd pick a quiet moment (or a sync boundary, if you're doing CRDT-style collaboration) and renumber the whole list with fresh, evenly spaced keys.
The library gives you generate_n_keys_between(NULL, NULL, n) which makes trivial to generate N fresh keys for the whole list in one call.
No, the library is intentionally minimal on that front. It generates and validates keys but doesn't track any metadata about them. Keys are plain C strings, so detecting "time to compact" is straightforward on the caller's side:
- Key length is the most natural signal, just strlen(). If your base62 keys are typically 3-5 characters and you start seeing 15+, that's a reasonable trigger.
- Max key length in the list is cheap to track if you're already iterating for display or sync.
Keeping that policy out of the library is intentional. What counts as "too long" depends entirely on the application. A SQLite column with a B-tree index cares about different thresholds than an in-memory CRDT. And the right moment to compact (sync boundary, idle tick, batch write) is an app-level scheduling decision the library has no business making.
One that lets me use it in my open source projects without then preventing other people from using my open source projects in their closed source projects.
Using your library currently completely disrupts the licensing situation for my own work.
Your site is for people familiar with SQLite. Imagine a person, who knows about vtable, wal, vfs, json1, fts5 and even blockcachevfsd, reading your page. It is between AI slop domain parking and a targeted phishing campaign. The first reaction is either to report scam or block spam. 90% of your comments here do not discuss your project but give assurance sqlite name is not being used unlawfully. Back when it was just vector extension, it at least made some sense, but this new web page is a complete assault on anyone familiar with sqlite internals. Please tune your messaging! Currently your prized "sqlite" trademark is being used like "with a computer", left and right.
Just to clarify a point that seems to come up in several comments: we are backed by SQLite’s author, and we have explicit permission to use “SQLite” in the company and product names.
We believe SQLite on the edge is a major opportunity, and our focus is on building genuinely new features and infrastructure on top of it.
I understand your concerns about the license, but our goal was simply to prevent large corporations from taking our work, forking it, and offering it to their customers while we struggle to sustain development. We need to monetize our work in order to survive, though we do offer very generous commercial licenses for those who are interested.
Why not release the source under AGPL (where 'network use' counts as distribution, unlike GPL), and offer commercial licences for those who want more favourable terms?
The maintainer of libxml2, Nick Wellnhofer, will be moving all his future contributions over to an AGPL fork as corporate users of libxml2 were unwilling to contribute financially.
The library does a few things to keep key growth practical:
But yes, in a long-lived list with adversarial insertion patterns (always inserting at the same spot), keys will grow unboundedly, that's inherent to the algorithm. In practice, compaction is an application-level concern: you'd pick a quiet moment (or a sync boundary, if you're doing CRDT-style collaboration) and renumber the whole list with fresh, evenly spaced keys.The library gives you generate_n_keys_between(NULL, NULL, n) which makes trivial to generate N fresh keys for the whole list in one call.
reply