How do you handle distribution of documents among nodes? Say document X has 3 tags "A", "B", "C". Do you place all documents with tag "A" in the same node, all docs with tag "B" in the same (but potentially different) node, and all docs with tag "C"...? In that case you'd need to put document X in potentially 3 different nodes. Or do you not do that kind of "pre-sorting" and just send queries to every node asking for match for tag "A"? (I like my username too. I was surprised it was still available)
Documents are distributed across nodes based on a primary key. Currently we use range-based sharding, but will be moving towards hash-based sharding soon. So, for any given document, we look at the primary key and determine where a document should be.
We store a secondary index for a shard on the same node where the master for the shard resides. So if you're storing users whose last names are between A-M on node A, all secondary indexes for users with last names between A and M will also be on node A. That means that for any secondary index query we have to contact all nodes that have shards for a table, but we do a number of systems tricks to make this really efficient.
