I've always been curious why someone hasn't (or why I haven't heard of someone) prototyping an improvement to MySQL's query cache to invalidate the cache less frequently and get more dependable usage. Currently it invalidates when any table referenced by the query changes. It seems like you could improve this algorithm without too much trouble. For example, it feels like if the query was located solely through indexes, it should be able to do cache invalidation on an update to those index hits rather than on the entire table.
I'll have to look through the code one day and see why this makes no sense.
Interesting use of the 20-second cookie. What happens when they scale to data centers in Europe, Asia, and South America? Does it become a arbitrary-second cookie depending on where the user is?
Nonetheless, I can appreciate how a simple approach solves the problem. If these are the most dramatic scaling issues that facebook faces, then this provides strong support to the argument that you shouldn't focus too much on scaling and optimization when first building your site. There are probably better uses of your early time.
Actually, these are really interesting problems. Multi-homing isn't an issue for most startups though (not that this is true multi-homing since their second DC is read-only, meaning that they can't really survive the loss of the primary).
But due to the replication delay, that would mean losing data on the primary, and a long delay on bringing the old primary back in sync with the new (since they have effectively forked). Therefore they would only switch the the secondary if the failure at the primary site was catastrophic -- it's more like a backup.
While there's a ton of talk about scaling on here, it isn't needed by the vast majority of people and even Facebook's way isn't that complicated when you get down to it.
Basically, the only change that needs to be done is that you have MySQL trigger the delete from memcached on replication. All your slave database gets an updated record for you, saves it in its tables and then hits memcached with a delete command for that key. While it does mean diving into the MySQL code, when you get to the size at which you need such a feature, you can just hire a MySQL expert.
This is probably a bad idea in a lot of cases and fb replication probably is rather complicated, most likely relying on some sort of system that relies on stale data when it can and asks for fresh data when it needs it. Number one if you're high scalability then you don't want to run a query when you already know how the underlying data has changed so any kind of invalidation trigger would be ill advised. Further, if you're replicating data over a large distance what's the chance that you'll ever have the right data in the right state. From the moment the data is changed on the master you probably need to fire off an invalidation, if not an update to that data, which is where some sort of collection class to manage this is probably more suitable. Invalidation will often quit to work well, even in a small replicated environment, if replication takes more than even a very small amount of time, which happens quite easily if a server is overloaded. Imagine I change my friends, wait on my process for replication, invalidate and some friend of mine views my page and re-caches my friend list from a mysql server that hasn't yet replicated my newest data. It's fairly solvable but even then you aren't sure that an update will go through on memcache so you have to have some sanity check to know whether you think the data is actually what you want to be seeing. I guess I don't really have a point but just more open ended problems for consideration for those of you wishing to scale. I think it's all pretty common sense but it's by no means simple. Best of luck.
The reasons there is a ton of talk about scaling here: a) everybody wants to start their own Facebook someday so they think they need to know this stuff AND b) scalability is actually kinda neat to study. Reorder them as you like but I think they are both factors.
That being said we are rapidly moving towards a world with more and more data being stored and queried. Scalability will therefore be something you increasingly need to know about in order to build a significant application -- regardless of your entrepreneurial ambitions.
Also, I disagree that 70ms makes that big of a difference. People are more used to slow sites then people realize - anything less then 200+ ms improvement in the aggregate and I wouldn't spend a dime (not to mention millions of dollars).
I'll have to look through the code one day and see why this makes no sense.