I would suggest that anyone trying a real comparison of vector DB's consider the following
- necessary functions / use cases (eg prefiltering, dense search)
- embeddings version management
- anticipated embedding size (the article only considers glove-100 on ANN-benchmarks, which is quite different from openai-ada-002 1536 - both in terms of their output distribution and the vector size)
- required precision / recall
- required ingestion speed
- required ingestion throughput / time to ingest periodic updates
- required query speed (percentiles, not average!)
- required query throughput
- required RBAC, data privacy, active-active, etc
...and so much more. ANN-benchmarks is a good start for thinking about this but remember that actual throughput is quite different from whatever you see in the algorithms benchmarking!
Wouldn't "ease of putting into production" also factor in?
For many use cases, being able to put a proof of concept out of the door in hours vs days vs weeks is the top selection criterion if everything else is "good enough".
dev experience is very very important, but I think it's so subjective -- if you are really pushing the boundaries, maybe you want a super powerful platform which can use all the bells and whistles, but if you want to hack a weekend project together, maybe you just want some API calls.
