Hacker News new | past | comments | ask | show | jobs | submit login

Dynamo's consistency ranges from fully consistent to loosely consistent and is tunable on a per-application and per-call basis. This means a write can be reconciled against 100% of the replicas before it is considered finished. How is this any different than the consistency guarantees an ACID compliant system provides?

You make a claim that Amazon uses ACID semantics for the checkout process. The Dynamo paper claims that in order to meet business goals, the complete Amazon.com order process must be highly-available and partition tolerant. A system that is ACID compliant must sacrifice availability or partition tolerance, but this isn't the case for Amazon's purchase process. Amazon simply strengthens consistency and durability guarantees in the case of checkouts with a quorum write. It would be exceedingly rare for a partition or disaster to knock out communication with more than one datacenter, so this works very well in practice.




Quorum writes to Dynamo nodes allow you make atomic and durable updates to single keys and the data associated with them. ACID transactions allow you to mutate data associated with multiple keys atomically. This is not the same thing at all. Common operations like debit from one account and credit another or sum a set of values become difficult.

Now atomic updates to single keys can actually be used as the building blocks for transactional functionality (using the lease pattern and/or compensating operations). If you need a transaction here or there, this might be a very workable solution. If you need lots of transactions, then you end up using Dynamo systems in unnatural ways; many of their performance and availability advantages are wasted in this configuration.

So yes, you could build a bank given quorum writes, but the point is that it would probably be a poor engineering choice.


I wasn't saying banks should use Dynamo, but that Dynamo-style consistency is appropriate for conducting e-commerce transactions, and originally that most scaled industries don't depend on ACID semantics in their core datastores.

Bank accounts are eventually consistent logs, which doesn't look like Dynamo, but also look nothing like an RDBMS table. It's well known and understood that almost all banking systems are based around mainframe-era batch-process systems that are eventually consistent in nature. It's awfully ironic that the "hello world" used to demonstrate transactions in the RDBMS world is a debit and credit of bank accounts, a situation that is vanishingly rare in the real world.

Also, please address my original rebuttal to your claim that Amazon uses an ACID database for their checkout process.


Again, replying to a deeper comment.

I sure don't disagree with everything you are saying.

As for adding transactions to get a balance, yes, this is how the basics work and it doesn't look like ACID at all. However, what you're describing isn't eventual-consistency (per Dynamo), it's eventual-completeness. You can't lose a log entry. All of these messages are two-phase committed between systems.

So as I said before, you could build this kind of system using quorum writes on a Dynamo system, but it wouldn't be a solid engineering choice.

Also, yes, BerkelyDB has multi-key transactions. Dynamo on top of BDB does not expose this functionality or use it (as described in the paper). Publicly available systems that implement Dynamo like Cassandra, Riak or SimpleDB do not have native transactions either.

Fun chatting. I'm off to bed.


Replying to the deeper nested comment.

1) I totally concede that you could use Dynamo-style consistency to be the system of record for financial exchanges. I just don't think it's a good idea. I'm not sure anyone building these systems does either.

Though if you're doing e-commerce with a payment processing gateway like PayPal, you might be in a different situation. You're no longer anything like a bank at that point (PayPal is).

2) I don't know what Amazon does now, but last time I talked with them (2009) they were huge users of legacy RDBMSs for actual order processing. I know they're not thrilled with this arrangement, but I doubt they think that they think Dynamo is the answer. Perhaps cores of ACID-ness with Dynamo-esque replication between them...

3) You're right that banks use logs for a tremendous amount of things. And a VISA charge looks nothing like a simple debit and credit transaction. However, if you think that banks don't have debit and credit transactions in their applications, you haven't worked on their applications.


I am starting to believe that we both essentially believe mostly the same points, but we're using different nomenclature.

You stated previously that most of these real-world, high-scalability, high-availability systems use ACID databases as a backing store, something I echo'd in the context of Dynamo's use of BerkeleyDB (and MySQL/InnoDB) as an underlying data store. I agree with this.

You state that a Dynamo-style key-value store wouldn't be that great for a financial institution's accounting system. I agree with this. Dynamo is terrible for "log" data.

This doesn't change the idea that these system as a whole aren't ACID, and do not require ACID semantics. Financial systems are often based on networks (in the conceptual sense) with varying degrees of trust, and are batch-reconciled logs of transactions. I'm talking about stock and commodity markets, electronic transfer systems, and traditional banking. They may be built from ACID building blocks at the low level, but that does not change the fact that systems that manage my checking account, credit cards, and stock market transactions are not functionally ACID or that they even need to be ACID. Dynamo itself is built on ACID databases, but it is not ACID itself.

The balance of an account can be calculated by adding all credits and debits, which is what makes a financial account radically different than say, a Facebook profile. Each transaction is essentially immutable. Even if the transaction must be reversed, it's reversed as an additional credit (or a unique reversal transaction), not a deletion of the original transaction. While transactions against the account are almost always immediately available, they aren't ALWAYS, and this is alright, because the bank's customers understand this. Banks still reconcile accounts on a nightly basis, using batch processes. This is when accounts are officially settled and consistency is applied, which is why it's eventual. However, financial accounts are somewhat unique because their date-sorted, log-structured nature makes them quite suited to eventual consistency.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: