Would you be willing to speculate on how VectorChord's ingestion and query performance might compare to Elasticsearch/OpenSearch for dense vector and sparse vector search use cases, particularly when dealing with larger full text data sets (>5M records)?
In the LAION-5M benchmark, we’ve compared our performance against ElasticSearch and OpenSearch. However, comparing ingestion performance is more challenging due to differences in architecture. Both ElasticSearch and OpenSearch, like most vector databases, use the concept of shards. Each shard represents a separate vector index, and queries aggregate results across these shards. Larger shards lead to faster queries but come with higher resource requirements and slower update speeds.
It’s also worth noting that ElasticSearch has implemented RaBitQ support for HNSW. So it's difficult to compare without running actual benchmarks. However, ElasticSearch typically requires at least double, if not triple, the memory size of the vector dataset to maintain system stability. In contrast, PostgreSQL can achieve a stable system with far fewer resources—for example, 32GB of memory is sufficient to manage 100 million vectors efficiently.
From my perspective, it would be faster in query comparing to ElasticSearch due to the extensive optimizations. And much much faster with the updates (insert and delete) due to using IVF instead of HNSW.
“There is no justice in the laws of nature, no term for fairness in the equations of motion. The Universe is neither evil, nor good, it simply does not care. The stars don't care, or the Sun, or the sky.
But they don't have to! WE care! There IS light in the world, and it is US!”
― Eliezer Yudkowsky, Harry Potter and the Methods of Rationality
I don't like this quote because it suggests a sort of separation between ourselves and the universe, or it suggests that the universe is some all-powerful omnipotence that might deign to heal us if only we could supplicate ourselves to it, but that's mistaken.
We're made of all the same stuff as those stars. We are the universe, and if we care, that means the universe cares as well. When we have the power to do something, that means the universe has the power to do something.
I was moved by your personal story, Irina. Thank you for sharing it with the world. I hope others who are struggling through similar challenges discover this article and find new hope and perspective in your words. I am happy you found your voice again.
Elastic, the company behind Elasticsearch and the ELK stack, is hiring Senior and Principal software engineers, Tech Leads, Designers, and Team Leads. Elastic is built upon a distributed, asynchronous working environment with over 1,900+ employees in 30+ countries. View our jobs and apply online at:
https://jobs.elastic.co/jobs/department/engineering#/
I recently found Checkvist for note taking. The progressive web app is really fast to load. The best part is extensive keyboard shortcuts for everything. Prior to finding Checkvist, I used text files for years.
Another idea I have heard is to encode your feature flags and entitlements as a bitwise string, and include this string in your customer's signed auth token. This way, your front end and back end don't have to hit your entitlements API all the time. When they change plans, just refresh the auth token.
