Hacker News new | past | comments | ask | show | jobs | submit login

Wow, saw the benchmarks:

https://docs.dalmatiner.io/en/latest/benchmarks/jpc.html

"8,500,000 and 9,000,000 metrics per second" on a 5 node cluster.

This is impressive.

The 1 node is impressive as well (and it shows how it scales):

1,500,000 metrics per second




They are impressive, but let's put some context:

  No guarantee of storage

  DalmatinerDB offers a 'best effort' on storing
  the metrics, the ingress transport is UDP and 
  there is no log for writes (there is the ZIL if 
  enabled in ZFS) or forced sync after each write.
  This means that if your network fails packets can 
  get lost, if your server crashes unwritten 
  data can be lost.

[1] https://github.com/dalmatinerdb/dalmatinerdb#no-guarantee-of...


Actually let me put that paragraph fully in context with the reality.

UDP is no longer used (this is outdated sorry for that), the connection is TCP now as it turned out over all the performance was better.

Dataloss can still occur since DalmatinerDB keeps a cache (which other metric stores might also do). A lot of that can be mitigated by using N=2 (or 3) to store data in multiple nodes that will reduce the chance of dataloss significantly. Keeping caches isn't uncommon however, to ensure full consistency it requires a kind of transaction from that goes from client to server to client, which is 'really' costly and I am convinced not worth it for metrics, a few seconds of lost metrics doesn't warrant the cost of that.


The tests were run with TCP, they do not include non stored metrics.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: