Given their start point (a product unfit for public consumption) that is the absolute minimum they need to do.
I at least will never trust Mongo for anything but a toy project. There are so many better options out there, options whose technical capabilities are as good as Mongo's marketing.
I think if you look back objectively, there are very few database platforms that were absolutely "fit for public consumption" right out of the box. Look at all the SQL Server shops out there (mine included) that won't even roll out a new version of SQL Server until it hits SP 1 at a minimum... For MongoDB, If you look forward based on what they are doing now rather than at how early adopters may have had a sub-optimal experience way back when, you'll see a mature product that is consistently improving and is demonstrably reliable. Can you give an example of another option you are referring to?
Right out of the box? Mongodb has been trying to get it right for 10 years now. Kyle says the storage engine they've used for most of that lifetime is fundamentally flawed, and they've only now, a decade on, managed to write something without known bugs to replace it. And maybe this time it's ok. Maybe this time there aren't any more layers of buggy crap in mongo yet to be found and fixed.
Maybe. But you'd have lost that bet if you made it any day in the last 10 years. And in those 10 years mongodb has demonstrated again and again that they aren't up to the task of writing a reliable database. Even with their new storage engine they couldn't find the bugs alone.
I think using mongo today for any mission critical data is an irresponsible choice. I'd seriously question the judgement of any senior engineer who picks it for a new project over rethinkdb or Postgres.
>"Kyle says the storage engine they've used for most of that lifetime is fundamentally flawed, and they've only now, a decade on, managed to write something without known bugs to replace it"
Didn't WiredTiger Inc write the new WiredTiger storage engine before they were acquired by MongoDB Inc?
Do you think MongoDB is a good choice (given how easy it is to use) when you only care that 99.999% of your data that you insert should end up in the database? That's my use case. Best-effort integrity. I mostly just want a DB can insert and query fast for documents and am not really concerned if I lose a few documents here and there.
It's about making tradeoffs. If MongoDB works for you (I actually enjoy using it tremendously) then I have to ask myself am I ok with its non-perfect integrity. For my use cases this isn't a problem. I'm not working with customer data or anything where losing a few records would make any difference at all.
How do you expect to check the end result? The article's Jepsen analysis shows that both the v0 and v1 replication protocols (excepting the very latest version of v1 that appears to be in response to this) can result in acknowledged writes being lost. I.e., the DB tells you, for a write sent with a majority, that the write was successful — to a majority! Subsequently (and, if I understand the article, possibly not immediately), the write can be lost.
Given a small cluster of reliable nodes on a reliable network, these errors will occur extremely rarely. So rarely, in fact, that they'll be written off as "user error" by support.
If you're a startup building a system which has to quickly and reliably scale from 3 > 3000 nodes in a year then the whole thing is likely to explode in your face. Twitter style.
Now, if MongoDB was so superior that it was truly platform which would even enable that kind of scaling, then the decision is simple: just go for it.
The thing is, this isn't how the world works. When systems are built, very few people consider (or are capable of considering) the growth of the system. Frameworks and database are, by the rule, chosen arbitrarily. When scaling happens, the question is more "how can we scale what we have whilst having everything kind of work" than "how do we design a system which works correctly at scale".
Mongo's whole strategy is based around this. Make Mongo the default choice for the current generation of developers.
Fantastic market strategy, but it's still snake oil they're selling.
When you talk about growing, the biggest value in Open Source has been that you can start with something free but shit, and then as you make money then you can spend it on customizing that Open Source in a way that benefits you.
However there exist commercial offerings that are (and were) faster and better at MongoDB than MongoDB was: KDB could've handled Twitter, we never would've seen a fail whale, and it is a whole hell of a lot cheaper than the developers and the customizers, and the headache, and the fact that you're making something open source which ultimately benefits your competition.
Another way to think about it is by thinking about experts: If you've got a great startup idea, why would you want to make your odds 10% worse by introducing the possibility it'll fail, by using the cheapest hacky hack thing that has 10% chance of losing your data? Ask experts with data, and be honest with your budget and you'll do a lot better.
I have some actual experience with KDB and MongoDB so I'm going to have to call bullshit.
How does KDB handle replication and failover? Or even high insert/update rates to datasets that exceed the size of memory? How do you shard KDB?
KDB doesn't support unicode text. Do you plan to only have English speaking users?
Yes, KDB excels at its relatively well defined niche of transforming and aggregating "smallish" (say 10 TB or less) numerical time series data. It would be a horrible choice for the backing store of a high throughput CRUD application...
What is it with KDB zealots thinking that KDB is the best database for every task? I swear, KDB is the Scientology of databases.
With -r and (in my case) SO_REUSEPORT. Most people use a dedicated gateway (have seen custom tomcat stuff and haproxy).
Meanwhile, MongoDB doesn't actually replicate reliably (acking then losing anyway) and failover can crash cascade in the naïve configuration.
> Or even high insert/update rates to datasets that exceed the size of memory?
This is literally the KDB tickerplant model. Have an RDB that flushes out regularly (daily) to an HDB.
You can also just write to a log `:log upsert ...
> How do you shard KDB?
Same way you shard anything else? By picking a key and directing the query to the appropriate server. h[(first md5 k) mod count h] "query..."
> KDB doesn't support unicode text.
UTF8 is fine.
The number of times I've needed the first 5 code points (and not the first 5 bytes or the first 5 characters) in my life is zero. All that half-baked Unicode support in various languages (like MongoDB) just makes people think that they've solved a problem that they really haven't.
> Yes, KDB excels at its relatively well defined niche of transforming and aggregating "smallish" (say 10 TB or less) numerical time series data. It would be a horrible choice for the backing store of a high throughput CRUD application...
I use it in one of those big CRUD databases (digital marketing and tele-lead tracking).
> What is it with KDB zealots thinking that KDB is the best database for every task? I swear, KDB is the Scientology of databases.
Because it solves problems they have.
Even when I don't use KDB I use a similar architecture because it's the correct architecture, because I've had these problems for a lot longer than I've had KDB.
If it doesn't solve every problem I have, that's because I have work to do, not because it isn't great at the problems it does solve, and I don't shout at my hammer because it isn't a spoon.
However MongoDB doesn't solve any problem I've ever had: I've never needed a bag of objects/filesytem that loses data, or a binary blob that I cannot query. It's so famously "web scalable" it has made a joke of the very idea of being scalable.
> With -r and (in my case) SO_REUSEPORT. Most people use a dedicated gateway (have seen custom tomcat stuff and haproxy).
So as a KDB user you need to implement your own HA solution. That is strictly worse than MongoDB replication, even with its now-fixed bugs. Do you really think your homemade multi-master KDB system would pass Jepsen?
> This is literally the KDB tickerplant model. Have an RDB that flushes out regularly (daily) to an HDB.
Wat? That only works if data is immutable once written. Tweets are liked/deleted/etc. You could store an immutable log of user actions, but then you would have to reconstruct the current snapshot every time someone loads a timeline. It's entirely possible for someone to like/delete/RT an old tweet. Financial data is naturally partitioned because the order book clears at the end of every trading day - this doesn't apply to CRUD apps.
> UTF8 is fine
I think you misunderstand what I mean by unicode support. Does KDB support locale specific collations? Does it support normalization/canonicalization? Being able to index by code point is about 1% of the needed solution to build an i18n-proof product. Obviously that doesn't matter when you are dealing with normal KDB datasets like market data where e.g. asian names are represented with numbers.
> I use it in one of those big CRUD databases (digital marketing and tele-lead tracking).
Were you using it to store clickstream data? Or some other kind of immutable stream of events? That isn't really applicable to general CRUD applications.
Like I said - KDB is great for analyzing immutable streams of events. It's not a general purpose database for building CRUD applications. MongoDB tries to be a reasonable enough solution for many use cases, while KDB focuses on excelling at a small number. Both are valid approaches to building a database...
> So as a KDB user you need to implement your own HA solution. That is strictly worse than MongoDB replication, even with its now-fixed bugs.
You simply cannot take MongoDB in its (near) default configuration, put it on AWS, and handle Twitter volumes.
I think "this broken tool is better than your working tool" represents a certain kind of madness that I can't argue with.
> Wat? That only works if data is immutable once written. Tweets are liked/deleted/etc. You could store an immutable log of user actions, but then you would have to reconstruct the current snapshot every time someone loads a timeline. It's entirely possible for someone to like/delete/RT an old tweet. Financial data is naturally partitioned because the order book clears at the end of every trading day - this doesn't apply to CRUD apps.
I don't know what your experience level is, but Financial data typically has many subscribers to that tickerplant, and build-up indexes an views representing the queries that consumers are actually going to be interested in. This is covered in the most basic of KDB tutorials[1]
When a user loads a timeline, ideally you want to hit a single machine on a single query to a machine near the viewing user. Processes representing tweet consumers subscribe to the ticker plant, and build-up the indexes of what information they're going to need to publish. You're also going to need to have a fast index-by-publisher as well, so that when a subscriber wants to follow someone, we don't need a replay -- again, more indexes, but at least these can be "centrally" located.
This isn't even a remotely difficult problem to solve with the right tools.
> I think you misunderstand what I mean by unicode support. Does KDB support locale specific collations? Does it support normalization/canonicalization? Being able to index by code point is about 1% of the needed solution to build an i18n-proof product. Obviously that doesn't matter when you are dealing with normal KDB datasets like market data where e.g. asian names are represented with numbers.
If I misunderstand you, it is because you are unclear.
JavaScript, C and C++ don't actually support "locale specific collations" even though there are well-maintained and well-distributed collation and localisation libraries that people can use.
That "iasc" doesn't know the difference between Chinese and American spellings for a word is irrelevant. I can solve the problems I have with my tools, and building sort keys on my symbol tables for each locale means that the user-visible aspects of sorting remain instantaneous, instead of being tricked into doing stupid shit like x.toLocaleString(user.getLocale()) which is slow at Twitter scale.
This is part of what I mean by "engineered correctly": The tools that are available to us can trick us into thinking certain problems are solved when they aren't.
> Were you using it to store clickstream data? Or some other kind of immutable stream of events? That isn't really applicable to general CRUD applications.
Tele-lead means (outbound) phone calls for the purpose of lead-generation, so I have phone calls and the results of those calls in KDB. It's not "big data" by any stretch of the imagination.
> Like I said - KDB is great for analyzing immutable streams of events. It's not a general purpose database for building CRUD applications. MongoDB tries to be a reasonable enough solution for many use cases, while KDB focuses on excelling at a small number. Both are valid approaches to building a database...
MongoDB is not a valid approach full stop: Build dogshit and then try to pepper over the bad press with "the new version isn't dogshit anymore" every few years is negligent at best, and pays dividends with the fact that it makes it easy to identify inexperienced engineers.
That KDB is not as accessible as MongoDB is Kx's problem, and not KDB's problem.
I replied to a sibling with more details. Indexing by code point is only the smallest (and easiest to solve) part of the problem of dealing with non ASCII text.
You check the result with getLastError which, as you described, can be used to ensure a majority agrees with the write. But you normally don't use getLastError that way. Because a majority might not even be concerned with that particular write. They are, after all, shards. Instead you check if primary got the write. If primary disconnects while you are checking, you catch the exception and try checking until a new primary is decided. And if your check result is not ok, you try inserting again. That's as reliable as it gets when inserting to any database including SQL databases that support transactions.
> I'd seriously question the judgement of any senior engineer who picks it for a new project over rethinkdb or Postgres.
... you mean RethinkDB, whose future is still uncertain? Regardless of technical merits, the currently unstable future of RethinkDB means a senior engineer should be extremely cautious about selecting it for a significant project.
To be fair, choosing a scalable database even for a senior engineer still requires quite a bit of very specialized knowledge in distributed systems that most simply don't have. So they have to rely on what "feels right" anyway, rather than making an engineering decision, and are very susceptible to all the marketing and PR and authoritative opinions. There are no right choices for them. Although if in doubt everyone should probably default to a dynamo-style db, as it forces you to think about and organize your data in a certain future-proof way, which actually excludes all of the mentioned databases.
Not the OP, but RethinkDB is superior in many ways including stability, integrity and the feature set for pretty much every use case you would consider using MongoDB.
But with Jespen tests MongoDB can finally be considered a contender. Its not like competent teams were using it in production. Right?
Do you not consider Stripe to be a competent team?
Please, name some F500 companies using RethinkDB to power critical infrastructure. There are many using MongoDB. While Rethink is widely renowned among the HN set it is nonexistent in comparison when looking at actual deployments.
The reigning HN view of MongoDB being a buggy mess is outdated. Yes, they overmarketed a buggy project in 2009. It didn't matter, because they built a product that developers loved (and continue to love) to use. RethinkDB didn't aggressively market itself, and look where it is now - defunct. Mongo used that momentum to raise money and hire an incredible engineering team, including Keith Bostic, one of the fathers of Unix, and Michael Cahill, the inventor of the transaction isolation mechanism used in Postgres. Sometimes you need to employ aggressive business tactics to get to a point where you have the engineering resources to build a world class project. Moreso when you need to catch up to millions of man hours spent building Oracle and MSSQL.
I should note that I work for a multi-national gaming company and we use software that is ABSOLUTELY not fit for purpose, but once you have a hard dependency on something and the cost of muddling through is _less_ than the cost of a rewrite then you're going to be stuck supporting it.
I don't think it should be taken as given that there's a correlation between competency and the size of an organization that a team exists within, and I don't think such a correlation, when combined with large organizations' usage of MongoDB would challenge the assertion that there exists an anticorrelation between team competency and use of MongoDB.
Looking at the numbers, larger organizations straight-forwardly seem like they should be more likely to eventually hire mediocre talent, survive despite having done so, and more likely to have adopted any given tool.
I think you're looking at it wrong. It's not a popularity contest; I've seen billion dollar companies use fucking stupid tooling as well, but they still have the right processes where they don't lose data.
In non-tech centric large orgs, it seems you frequently do not have the talent required to be be both risk averse and productive at the same time, so "IT" becomes a risk averse and non-productive political structure from which springs an "alternative IT" rebellion group (if the initiative is lead from above) or many little cowboy teams (if the initiative is driven from below) and these will be "productive" at the expense of having no processes for avoiding stupid risks that, amongst other things, can lead to data loss.
The point? How do you draw a line from "Fortune 500s use Mongo" to "Mongo is used by competent teams"?
I specifically challenge that you can do so, given a model of a large organization as being necessarily more diverse (regressing to the mean of general competency, more likely to have facts like "org Y uses tech X" being true) and more robust to survive failures as it grows.
We don't have data, but we can still model (if nothing else, to think about what data we would need).
> Its not like competent teams were using it in production. Right?
I've heard that about 1000 times a day for 6 years. Usually the person stating it is snickering as if they are clued into some unknown secret.
Mongo does work in production at many shops, and in many forms. Sometimes it's used as the main database, sometimes it's used to house specific slices of data, etc.
As a sysadmin, I got tired of the devs constantly ragging on Mongodb (the same folks that selected it before I was hired). I eventually got fed up and said: "why do we use it if you all hate it so much. Let's replace it. What do you want to use instead, it's easy for me to set up something new". Cue everyone going "ah, it's not so bad, really..."
MongoDB is the Nickelback of databases: a reasonable act that's not going to blow your socks off, but one where saying "OMG I hate it!" somehow signals membership to some cool clique of connoisseurs.
alternative anecdota : I rewrote a backend that was using mongo, moved it to postgresql+postgis. Solved an ever-expanding RAM issue and is still blindingly fast [ on SSD hosting ]
The main win was not server stability, it was having general tools to manage data .. including the inbuilt geo-algorithms that come with postGIS. eg. I could make our data set 9x smaller by smoothing map paths.
I still love the Mongo api, but I just cant risk it with data on projects that people are paying for, or that I need to support.
[ I think the real sweet spot will be deep integration of javascript and json into postgres - so I can write stored procs in js, get db events in js, wrangle json fluidly.. all of which is improving. ]
>OMG I hate it!" somehow signals membership to some cool clique of connoisseurs
You do realize you are the one dragging identity into the mix.
I dont understand how this is a big ego debate. MongoDB isnt that relevant and it isnt Nickelback, its a homeopathetic database. That is: if you use it for something else than caching (store data you cant afford to loose, or load balance and use the database as main mutex to deal with all concurrency issues) that would arguably be a very irresponsible choice.
Hell i hate most databases, because its hard to get right yet some have interesting trade-offs (ElasticSearch, Cassandra, CockroachDB).
And its not a subjective or even analog discussion where databases are more or less consistent or more or less durable. They fsync or they dont. They use raft with majority consensus or they dont.
If you as a sysadmin judge these emperical facts based on your prejudices about the sort of people that would agree or disagree with you than you are much more like the cool clique of connoisseurs than the people at the other end of your finger.
Its engineering, not wine tasting. The shape of the world isnt a subjective thing anymore than the durability of a database that doesnt fsync.
Well, I'm a member of the MongoDB is worth quitting over club. I replaced it at a former employer with postgres. Eng waved bye to an endless stream of operational issues, and customers saw better uptimes and much much faster responses.
Rethinkdb is still the new database on the block, and never really found it's feet. Look up Postgres, Cassandra, Kafka, Riak, MySQL/Mariadb or MSSQL. (Or dare I say it, oracle). All of those tools have a long history of reliability and solid engineering.
You're comparing ISAM/MySAM (storage engine) to the MongoDB replication protocol. As a more relevant parallel MongoDB also replaced its original storage engine with one acquired from WiredTiger (BerkeleyDB founders).
One big difference from a corporate strategy perspective is that MySQL let the replacement storage engine (InnoDB) fall in to the hands of Oracle. MongoDB was smart enough to make sure that they were the acquirer, which puts them in control of their own destiny.
If MongoDB is heading along the path of MySQL, that's a pretty good path to be on considering that MySQL is used as the store of record at Facebook, Twitter and some parts of Google.
> You're comparing ISAM/MySAM (storage engine) to the MongoDB replication protocol. As a more relevant parallel MongoDB also replaced its original storage engine with one acquired from WiredTiger (BerkeleyDB founders).
My bad, MMAPv1 vs WiredTiger although I think it was obvious what I meant.
> One big difference from a corporate strategy perspective is that MySQL let the replacement storage engine (InnoDB) fall in to the hands of Oracle. MongoDB was smart enough to make sure that they were the acquirer, which puts them in control of their own destiny.
Not sure if that's relevant though, since whole MySQL became property of Oracle (after they acquired Sun).
Yes... its very relevant... If MySQL had acquired InnoDB instead of Oracle there is a good chance they would still exist as an independent entity. That's what I meant by MongoDB's acquisition of WT putting them in control of their own destiny.
> Can you give an example of another option you are referring to?
That depends on the data.
What type of data you have and what you want to do with it.
MongoDB isn't data specific and it claim to fame is flexible data structure.
If you want fast write and look up with very little relation Cassandra is good.
If you want searchable text document then anything that is base on Lucene is good (ES, Solr, Raven).
If you want time series there are few out there but it's a niche.
Likewise if you want graph data then there are NodeJS, Titan, etc..
MongoDB at most company I worked with was use because they don't think about what type of data it is and what performance they want. They want to store unstructure data cause it's easy.
I personally think it's a cop out, especially as a statistician/programmer.
What if I want filtering by several criteria (on a table with 1k columns) and simple aggregations, but I don't need full-text search* ? I'm still looking :(
* I only want starts-with and contains on strings.
SQL Server and Oracle are hardly comparable. The issues you see there are more to do with backwards compatibility or performance impact of big new features than they are with the core stability.
> As far as I can ascertain, RethinkDB’s safety claims are accurate. You can lose updates if you write with anything less than majority, and see assorted read anomalies with single or outdated reads, but majority/majority appears linearizable.
The safety claims are accurate but like all databases, its never 100% perfect as there are tradeoffs.
> That reordering was only possible because of a special workaround for a bug in an older version of RethinkDB.
> What are the risks to users? The RethinkDB team and I suspect it’s unlikely this bug will occur outside of stress testing. Cluster reconfiguration is typically infrequent, and users would need a specific series of network failures or other message delays which happen to cut the replicas apart—in a way which allows both network components to find independent majorities for their respective table configurations. In Jepsen tests, it usually takes tens to hundreds of partition/reconfigure rounds to trigger this bug.
This isn't really an architectural failure (as you see repeatedly with MongoDB) but an implementation bug.
Compared to the failures in MongoDB that are frequently architectural (rather than implementation bugs), RethinkDB passed easily and quickly fixed the implementation bugs.
Rethink and Cassandra both fare far better in Jepsen.
Depending on the exact requirements Postgres (in the right configuration) and Redis are better options.
I'd argue that for 90% of MongoDB users - the group whose requirements were met by MySQL/MyISAM - would be better served by Postgres or MySQL with a proper storage engine.
Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. There were also a few implementation issues around LWT (Paxos) but at that time Paxos was very new. There were no architectural / fundamental changes needed to correct these issues. Also, even before fixing, Cassandra was losing only a tiny fraction of data than Mongo was losing. So, yes, this is what I call "far better", even if not initially perfect.
Pretty sure Cassandra still uses local timestamps and last-write-wins, which means you can't do safe updates. Also pretty sure they didn't fix the timestamp-collision row-isolation issue I talked about, because it was on HN's front page a few months ago. ;-)
Using timestamps and last write wins is a well known and well documented behavior, so you must assume updates can be sometimes applied in a different order than they were submitted. However, reordering updates is not the same as rolling back writes as was in case of Mongo. Cassandra does not promise linearizability in this mode of operation (if not using LWTs), and this is a tradeoff to get better availability. There is no better way than last write wins if you want high availability and partition tolerance and don't want to pay the performance and availability price for a consensus algorithm like Paxos or Raft. And no, vector clocks do not solve this problem in practice at all.
As for the post on HN from a few months ago, I remember only one guy who mixed LWT and non LWT updates and was surprised with lack of linearizability. Not a Cassandra fault if somebody doesn't know what he's doing.
> assume updates can be sometimes applied in a different order than they were submitted
I suggest you re-read the analysis. Some databases can offer safe, generalized commutative updates; e.g. Riak. Cassandra can't: updates, in general, can be lost through reordering.
> There is no better way than last write wins if you want high availability and partition tolerance and don't want to pay the performance and availability price for a consensus algorithm like Paxos or Raft.
You can have safe updates through clustering columns or if you really insist on destructive updates - through LWTs. With clustering columns you can easily achieve whatever is possible with vector clocks.
As for the research you posted, there is no free lunch. Each of these strategies come with their own set of drawbacks. That's why Cassandra offers choice at a query level.
Clustering columns do not make Cassandra updates any safer: they only reduce the scope of conflicts. It's still last-write-wins--the approach that Cassandra's own blog recognizes as "a high potential for data loss".
That's why Cassandra offers choice at a query level.
It doesn't give you a choice: you get last-write-wins, or some limited merge functions with CQL types. You can't get generalized CRDTs in Cassandra, because they won't expose conflicts to the user layer. Cassandra gives up on a whole class of safe AP datatypes as a consequence of this restriction.
Now you are arguing about a lack of a particular builtin feature. Riak is last-write-wins by default with optional vector clocks and builtin CRDTs. Cassandra is last-write-wins by default with clustering columns allowing to implement an equivalent of Riak vector clocks and CRDTs in the application. If you include a client id in the primary key and use client-side timestamps, you essentially are doing vector clocks and there are no conflicts guaranteed - users of Cassandra have been doing it for years.
We built examiner.com on MongoDB in 2010. Yes, that was before the journal was introduced. Would you call a then Quantcast Top 100 site a "toy project"? It served well into 2016.
I said that I would only use it for a toy project. Or at least a project for whose requirements resulted in MongoDB being a good choice. For example a content site where losing some data doesn't matter and whose hot datasets would never expand above the memory available by my budget.
Given the projects I chose to work on, MongoDB would be a bad choice. So I would only use it for a toy project right now. In 3-5 years time I expect that it will be a very good choice, given that they've pretty much won and can hire all of the good engineering talent.
The parallels with MySQL are really astounding; I just hope that they don't end up being sold to Oracle or Microsoft.
I think you're going to end up on the wrong side of this one, if you're not willing to change your opinion based on new info. It's a big deal, his second to last sentence in the intro:
> While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite.
I'll change my opinion once they've actually fixed it. The same way my opinion of MySQL changed when it finally matured (around 5.0 when they added proper transactions).
And big news for me is the existence of that Evergreen tool as an open source project. Looks great! Is anyone else using it? Is there a comparison to Jenkins available?
Not exactly, but we published a blog post about why we took the time to write a new CI system, and that post does touch upon why we didn't use Jenkins + plugins.
MongoDB receives a fair amount of criticism here but the company is a fantastic place to work. I'm proud that I was able to learn and grow as a developer alongside all of those who have been trying (and succeeding) to make a great database.
The team at MongoDB really cares a lot about making the best database product possible. I knew it when I was there and still think so after I've left.
MongoDB as a product evolved tremendously in the last few years. All the improvements introduced with WiredTiger and other major releases have seriously improved the performance and reliability of the product. It's not a perfect solution, but the majority of criticism is something that was written about version 1.8.
Actually, having been through their training, and dealt with their consultants. Their company culture is the problem. They had a pure-developer centric mindset, and no operations mindset. So at the time they had no good way to do backups. Their more modern solution is backups that they manage for you, which is even more crazy.
Yes. Mongo is an operational nightmare. I mean, who in their right mind would use a product where the recommended solution[0] to resync a stale replica (which happen all the time after a long netsplit) is to "perform an initial sync". Which, of course, mean "please remove everything and type this command". Crazy.
Indeed, and the only practical means of performing a compaction is to "rm -rf" the data directory and let it resync from another replica set member. This is not documented of course.
This is documented (okay, a little "hidden")! It's written black on white in their documentation (see the link above):
> A replica set member becomes “stale” when its replication process falls so far behind that the primary overwrites oplog entries the member has not yet replicated. The member cannot catch up and becomes “stale.” When this occurs, you must completely resynchronize the member by removing its data and performing an initial sync.
> MongoDB provides two options for performing an initial sync:
> Restart the mongod with an empty data directory and let MongoDB’s normal initial syncing feature restore the data
> Restart the machine with a copy of a recent data directory from another member in the replica set.
Note: the second option is not a real option when you're dealing with a 700GB database. By the time you finish the copy the oplog will be too big anyway. Thus, making all these steps completely pointless.
And that's why it's so bad. They even acknowledge the "correct" solution is to rm your data and resync.
It's documented for the use case of "stale replica" but I was referring to the use case for when you want a compaction to reclaim disk space. For that they recommend the db.repairDatabase() option but that requires you to have twice the size of your db available available on whatever partition your database is on. That was I said "practical." But yes the procedure is the same.
It's been a long way from the "Call Me Maybe: MongoDB" post from years back. Aphyr/Kyle took them to task in so many ways for playing fast and loose with data integrity, and rightly so. MongoDB could have said, "that guy's full of BS, ignore him," but instead they did the smart thing and paid Kyle to help solve the problem.
n.b. I can't find the original "Call Me Maybe" post, but this later one [1] is similar.
I didn't say Kyle solved Mongo's problems, but that he helped solve the problems. He identified numerous data loss issues, explained them in great detail, and even has tooling to readily reproduce the data loss events. That's a massive contribution in my mind.
By comparison, it's the difference between a bug report that says "you lost my data, you suck!" and "I've identified [x] ways your product can lose data and here's exactly how to reproduce each one."
I'm not exactly sure what you're trying to say, but perhaps I can help provide context:
In 2013, I performed an unpaid analysis, in my nights and weekends, of MongoDB. I found a bug leading to the loss of acknowledged writes with majority write concern. Mongo fixed this bug within a few weeks.
In 2015, I performed a followup test as a part of my work at Stripe. I confirmed dirty reads (which were already documented, though perhaps not well-appreciated), and discovered stale reads (which ran counter to MongoDB's documentation). MongoDB wasn't enthusiastic about that report initially, but got things sorted out and started work on adding majority and linearizable read concerns. I found that writes appeared linearizable.
In spring and summer of 2016, MongoDB paid me to help expand the Jepsen tests and hook them up to their internal CI system, so they could use the Jepsen tests to help verify their ongoing work towards linearizable reads. I privately confirmed that MongoDB still failed to prevent stale reads, but the dirty-read failures I'd seen in 2015 appeared to be prevented by majority reads.
In fall 2016, MongoDB announced they were almost ready to release 3.4.0 with support for linearizable reads, and paid me to perform a full analysis so they could be more confident in the results. It passed the linearizability tests I had initially written in 2015, but I offered to expand the tests to be more aggressive. Our collaboration resulted in the present analysis, uncovering design flaws in v0 and implementation bugs in v1. MongoDB worked to develop patches prior to 3.4.0's release, and that's why it passes now. :)
So a client can get the analysis done and leave it unpublished if they don't like the results? Maybe that's standard practice but it doesn't seem great for users.
I agree. In my ideal world I publish everything immediately.
There's no "standard practice" that I know of--very few people are doing this kind of work. Behind every one of these analyses is weeks of contract negotiation where I try to convince assorted lawyers & CFOs to go along with my weird, idealistic ethical policies. And conversely, those lawyers and CFOs do their best to balance the desire for correctness with their duty to protect the company. This policy outlines how we compromise.
Originally I did offer a client veto, because clients weren't willing to sign without some assurance of control over the outcome. My current standard contract for analyses actually drops that client veto: I have final say on the analysis content and publication. There's still a grace period to allow the vendor to fix bugs & get things in order. I'm adopting this as the standard going forward, but it's been a long road to get there.
That said, I do perform private consulting--usually for in-house systems, sometimes for clients that can't afford a full analysis, and sometimes as a precursor to more involved work. That means that yes, vendors may be aware of bugs and I won't have told you about them--but I promise that my public work remains honest and forthcoming.
Hey, just wanted to say I really appreciate all the in depth posts you put out on your site in hopes of educating others and helping them navigate the waters of distributed systems.
Curious, is there any single book you would say you've found to be the equivalent of say CLRS but for the fundamentals of distributed computing paradigms?
My current plan for deep diving into distributed systems theory has just been to go through resources like "Distributed systems for fun and profit", and aggregated lists/overviews like https://henryr.github.io/distributed-systems-readings/ and your distsys-class notes and then just DFS into certain topics from there - compared to the approach I've been taking with deep diving into OS and database theory which is just to go through text books like Operating System Concepts and Database System Concepts while applying the theory in side projects.
> "Once I've prepared a written analysis, the client may defer publication by up to three months. This gives the client a chance to understand the faults I've identified, attempt to fix bugs, update documentation, and deploy fixes to users."
So, a client can get the analysis done and attempt to defer publication for three months, after which time it'll be published. No?
MongoDB 3.4 passes the rigorous and tough Jepsen test. Jepsen designs tests to make databases fail in terms of data consistency, correctness, and safety... MongoDB 3.4 passed through their newest tests.
I think that this really shows how mature of a Database MongoDB is.
While you are correct that 3.4 now passes, I feel like your interpretation here is a bit optimistic. It's not like I tested Mongo and it passed out of the box--it failed Jepsen tests, and not just with a read anomaly--it lost majority-acknowledged inserts. The v0 protocol still fails--it's fundamentally broken. The v1 bugs are fixed now, but that's a consequence of our collaboration.
I'd say the marker of maturity here is that MongoDB has put significant time and effort into correctness: they take clock skew and network partitions as serious failure modes, they've redesigned their replication protocol, added options for stronger reads, and invested in their own correctness test suite, and Jepsen tests, as a part of their CI process.
I'd like to use the occasion to thank you for the service you provide to the community. The tests themselves, the collaboration with vendors, the competition it fosters and the in-depth technical write-ups I think push the whole ecosystem forward, which everyone eventually benefits from. Much obliged.
I agree -- I think the two most important people currently in software are aphyr for database testing, and lcamtuf for fuzzing (AFL). Both are pushing important fundamental software to pass the kind of tests it's embarrassing we haven't been doing for the past 20 years. I mean that in the most positive way, it's hard to get people to do things they've been neglecting for so long!
If you look at past Jepsen test mostly they fail, and sometimes they fail spectacularly. This is one of those things where if you don't have an automated test suite to continually verify correctness you will end up with bugs.
> In addition, we show that the new v1 replication protocol has multiple bugs, allowing data loss in all versions up to MongoDB 3.2.11 and 3.4.0-rc4. While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite.
Wait, it is me not understanding what the abstract is trying to say?
It's saying it was fixed in the final 3.4.0 release (or, maybe, that there's at least a patch available) for the v1 protocol, but that the v0 protocol is unfixable - so, 3.4.0 + V1 protocol and it's solid.
Giving aphyr some money to do this makes me think much more of Mongo now; I wouldn't previously have considered it for serious use but this is an excellent demonstration of intent.
I can only think that the article was a bit late and the new MongoDB version has the last word. Beyond this, the article is extremely valuable and it is mandatory to upgrade your MongoDB.
This analysis was a collaboration with MongoDB over the past three months. MongoDB was able to fix these issues in 3.4.0 because I found them in rc3 and rc4.
It's great that they're serious and committing resources to passing these tests. They just did, though, after nearly a decade of development. Which is terrific but it doesn't really show the 'how mature of a database MongoDB is'.
It's one test, paid for by MongoDB. It's good that they work with the dude. But there's only one test in town.
I want to see more than just one test from more than one source and seeing their sketchy performance in the past only time will tell if this new version will be any better.
> I think that this really shows how mature of a Database MongoDB is.
Yeah no. It's a test. We have to wait and see in productions and see the data to prove that. It's also the first version to past this test. It means something but not a whole lot without real production data and other people attesting to it.
I think this is 10th year of MongoDB's existence. This year they seem to be having not just popular but good product. It will be interesting to see if ratio for Good/Popularity is similar for most non-traditional databases.
Good news to hear!
We just finished a fairly big IoT project where we needed to ship a replicated database which was fast, easy to deploy and reliable. We tried other DBs but none was so stable to be shipped with a standalone application completely black boxed to the customer. I was septic in the beginning but after 800 installations in 2 month we have no complains. Also not from early adopters which run now for appr. a year.
Sorry if this is a stupid question. But how important is guaranteed integrity, in practice? (BTW, is what you're referring to considered transactions?) Obviously, for things like banking you need guarantees, but I'm not sure how often reads/writes to mongodb just disappear out of thin air.
>but I'm not sure how often reads/writes to mongodb just disappear out of thin air.
Which would imply you have to restore from backups continuesly. Or else any bug you run into might be the result of data corruption.
Its the fact that you cant even tell. And its not just about loosing data. If you rely upon a consistent world view (i.e. schedule payment, jobs scheduling, vms provision or shipping, account systems) the risk is unacceptable.
This is great news. Historically I've accepted the risk of data loss and coded checks when needed. I will never rely on my database, regardless of which one I am using for complete data consistency. It is, however, nice to see strides being made towards even better robustness. Go MongoDB!
> I will never rely on my database, regardless of which one I am using for complete data consistency.
I can't imagine developing any software that involves relationships between entities that does not have data consistency. Check constraints, foreign keys, and data type validation all provide a minimum sanity level of the underlying data that allows your mind to focus on more important things. Otherwise you're entire application is going to be littered with those same sanity checks.
I'm not saying that all apps need that type of data store or that there isn't room in this world for NoSQL stores, I mean specifically that complicated interdependencies and validation checks lend themselves well to the relational model.
I'm sure they exist but I can't imagine any project where I would want to factor in data loss from my database.
At the very least I'd want to know exactly how much data loss we are talking about, which is probably harder to answer than just using a platform that doesn't suffer data loss in the first place.
I see many people say that Mongo is unfot for a production db, but it is actually used in production by many companies.
Obviously this is true of many products, but how serious is it in practice? The criteria for 'good enough' can't be passing Jespen because very few db's do.
Storing schemaless json is a very valid use case. And the fact is there aren't really many proper NoSql alternatives, other than RethinkDB, which I hope becomes popular, and maybe Couch. e.g. Cassandra is often touted but its a k-v store, not a document db.
There's a reason Mongo became and continues to be popular. I still think Rethink is superior in every way.
I know there are a lot of bad vibes on HN about MongoDB, but I also know several companies which MongoDB enabled with the right libraries to much faster iterate on "schema" and business requirements (in their growth phase) than with other databases at that time (some years ago). The schemaless approach of MongoDB suited their fast changing needs much better than e.g. RDBMs.
What currently still makes MongoDB nicer than e.g. Postgres JSON are the libraries embracing schemaless while JSON in PG libs still feels tacked on.
Recently I had some bad experience with MongoDB support though.
Protocol v1 should be largely transparent to clients (at least for standard kv operations). It's a change you make on the server, at the replica set level.
These 'from the ground up' totally all-new-code approaches to DBs are just a scary proposition. Think of the thousands of man-years of effort that went into building MySQL, testing its codebase, and perfecting it's robustness (fail-proof-ness). What does MongoDB bring that couldn't have 'built on top' of MySQL codebase, and used MySQL transational layer as it's underpinnings. Sure, MongoDB gets all its performance gains from delaying writes to the DB (eventually consistent), caching in memory, and dispensing with ACID, however there is nothing about MongoDB that couldn't have been written to use the DB layer of MySQL at the point where it actually does its writes to disk. In this way, MongoDB would have revolutionized the world rather than mostly "fragmenting" the storage space.
I guess there are those who will say that even using batch-commits, that MongoDB could never have achieved the performance it currently does (by bypassing any ACID) if it was built on top of MySQL. But regardless, why not focus efforts on improving MySQL batch processing performance rather than throwing it all out, starting from scratch, and writing directly to disk. I know MongoDB became a success, but I think that is 'in spite of' their decision to start from scratch and not 'because of' it. Also think of the power that would be available if there were some relational table capability (true real MySQL ACID) right inside MongoDB whenever it was needed, if they were the 'same animal', rather than having to use two totally and completely separate DBs if you need NoSQL and also ACID in an app, which 99% of apps DO NEED, at some point, once an app grows beyond the round-one funding startup-toy phase and MongoDB falls flat in it's RDB capabilities.
> Also think of the power that would be available if there were some relational table capability (true real MySQL ACID) right inside MongoDB whenever it was needed, if they were the 'same animal', rather than having to use two totally and completely separate DBs
Just use Postgres (or several other options) then? It has all that built right in if it's what you need. But...
> if you need NoSQL and also ACID in an app, which 99% of apps DO NEED, at some point,
...I doubt 99% of apps really need this.
> once an app grows beyond the round-one funding startup-toy phase and MongoDB falls flat in it's RDB capabilities.
Or don't try to force MongoDB to behave like a RDB and you won't hit those problems? I just moved a system from Postgres to MongoDB, and it's running faster on way cheaper hardware now. Not because either database is inherently better than the other, but the use case lined up with mongodb perfectly and the old model was leveraging Postgres really poorly. Eventual consistency is fine, and I can denormalize certain data for faster reads because I know it won't be modified later.
I don't know Postgres, and I just consider MySQL to be the leading best-in-class open source DB engine available. I realize that's debatable. But personally my mind is made up on that.
I think relational-type queries are so common that any significant app will need them. For my own side project (meta64.com) I'm using JCR-SQL2, on Apache Oak jackrabbit, so I get to do lookups fairly easily, but it sure would be nice to have a full-blown ACID RDBMS engine sitting right there to use also, in the same engine.
What does MongoDB bring that couldn't have 'built on top' of MySQL codebase, and used MySQL transational layer as it's underpinnings.
There are two challenges here. One is that MongoDB has a different data model than MySQL: hierarchical, schema-less documents instead of uniformly-typed flat tables. It's possible to map one to the other--look at Postgres' support for JSON datatypes. That involves extensions to both the storage format and query language, but it's certainly doable.
The bigger problem is that MySQL's local transactional isolation is nice, but not particularly helpful in a distributed context. Distributed transactions--even single-document ones--are still tough to implement correctly and efficiently. As an example, consider Percona XtraDB, or MySQL+Galera Cluster. Both are building on a serializable single-node system--but in a distributed context, they wind up failing to provide snapshot isolation, let alone serializability.
The point is that MongoDB writes/reads directly to File system. It could have been (and still could be retro-fitted) to ride on top of an RDBMS for all it's actual IO, and create a hybrid. I'm aware of all the orthogonal points you raised which have nothing to do with ruling out RDBMS as I/O
Probably because there has been a lot of development in distributed databases in the last few years. Raft is fairly new, for one.
Taking really old code and trying to make it do new things is very very hard. After you've invested all that time and effort making mysql better, it's still a database owned by oracle.
Think how Java would have turned out if Sun made it to be cross compiled to C rather than run in a VM.
I haven't seen the codebase of MySQL, but it's open source and not "owned" by Oracle. The only reason anyone should want to start over is if that codebase itself was crappy code. I bet it isn't. Implementations of a robust RDBMS takes decades. MySQL is very mature. That doesn't act against it, it acts in favor of it. The better analogy is Linux. I'd be more likely to say "Linux is bad because of age" long before I'd ever say "MySQL is bad only due to age."
There are plenty of examples in the Language world also: Go, Rust, etc. All those new languages SHOULD have built on top of Java, for the same reason Mongo should have built on top of MySQL. BTW, Java DID leverage C++. The Java compiler IS written in C. Mongo didn't use any part of MySQL, but Java uses EVERY part of C.
Yeah, you are taking that sentence too literally. What I mean is we don't need a new language created every week. Java is already perfectly fine, and IF somebody finds something Java can't do, they shouldn't start from scratch and create a new language, but instead build and improve on what already exists: Java. It's like you can either stand on the shoulders of giants or you can start out at dirt level. I don't like starting back in the dirt. Not to mention the fact that every time somebody reinvents an already existing wheel, they just fragment the industry, because half the script-kiddies out there (not knowing any better) will simply jump on the bandwagon of whatever language most recently went viral. The Sisyphus approach to technological advancement.
Open and available for everyone to see, for every build of MongoDB. Is there another database that has this much transparency? (for every build)