No guarantee of storage
DalmatinerDB offers a 'best effort' on storing
the metrics, the ingress transport is UDP and
there is no log for writes (there is the ZIL if
enabled in ZFS) or forced sync after each write.
This means that if your network fails packets can
get lost, if your server crashes unwritten
data can be lost.
Actually let me put that paragraph fully in context with the reality.
UDP is no longer used (this is outdated sorry for that), the connection is TCP now as it turned out over all the performance was better.
Dataloss can still occur since DalmatinerDB keeps a cache (which other metric stores might also do). A lot of that can be mitigated by using N=2 (or 3) to store data in multiple nodes that will reduce the chance of dataloss significantly. Keeping caches isn't uncommon however, to ensure full consistency it requires a kind of transaction from that goes from client to server to client, which is 'really' costly and I am convinced not worth it for metrics, a few seconds of lost metrics doesn't warrant the cost of that.
https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html
"8,500,000 and 9,000,000 metrics per second" on a 5 node cluster.
This is impressive.
The 1 node is impressive as well (and it shows how it scales):
1,500,000 metrics per second