Web or no web, the fastest way to become an overnight success is to put in ten years of hard work.

I think his point is that the web is no different from any other business, even if it looks like things happen much faster, in reality you have to keep working for a long time before things really go big.

Of course, I might be wrong :-)

I would tend to disagree with that strategy.

First, memcached is fast. Allocations are made in O(1) time (think: scales infinitely) and is non-blocking. Anytime you want to look up an object, you just call for it.

With sharding, the process is more difficult. First, you query for the shard of the object, then you query that shard. OK, not too bad. What happens when you want to get all the friends of Adam? Well, you query for Adam's shard, get the list of his friends, then query for each of those friend's shards, then query each of the friend's shards for their data? Ugly, slow, gross! OK, so you denormalize! Only problem with that is that you then have to do a lot more queries on writes (which are the hard part to begin with). Plus, what if you want to search for users named 'Bob'. Well, typically sharding involves a setup like table, pk, shard - to relate an item to which shard it's on. If you're looking up by the primary key, you're gold. Not so much on the other fields. Yeah, you can query each shard, have it send back results and combine them yourself - in fact you could even automate that process in a proxy - but it isn't the most wonderful solution.

That scenario in memcached works much nicer - mostly because memcached is basically a sharded database to begin with. You don't have to look up which memcached server an object is on. Just pass it the ids you want ('person:5', 'person:22', 'person:900') and it will grab them from the appropriate shards and send them back! The problem is that memcached doesn't keep multiple copies so it can't be used for persistence. No problem, MySQL will handle that!

What's really missing from the whole debate is that when you get to a certain point in your application, you just don't get to query like you used to. With memcached or sharding, you're going toward limiting yourself to keys - somewhat similar to Google's BigTable (a dumb store).

With AppEngine, Google took a huge step to making this more usable to programmers. By adding an index on each field (with the option for indicies on multi-field lookups), AppEngine allows you to have a sharded database without sacrificing some querying capability. So, the big question is, why isn't someone developing a system that works like that - spews your data all around, but keeps an index available for some query capability? In fact, AppEngine's datastore queries work shockingly similar to views in CouchDB - look at how scanning a range is done in each and you'll get an idea.

In the meantime, MySQL/memcached in combination allow me to take advantage of both sharding and querying with relative ease. So, I save Bob as a friend of Adam in MySQL, then I update the entries in memcached for each of them by querying MySQL for their records and friend_ids and put {'id': 12, 'name': 'Adam', 'friends': [1, 212, 78, 51]} and {'id': 212, 'name': 'Bob', 'friends': [12, 491, 51, 999]} into memcached. When I want Adam and his friends, I run two memcached calls - give me 12, give me (1, 212, 78, 51).

That isn't perfect, but it can be a lot easier than sharding. At some point, it becomes requisite to shard, but it's not fun so why not use memcached along the way? "Don't prematurely optimize" comes to mind.

Yo. If you give me your email address, I can give you an invite.

As for your questions, the short answer is that we're holding back on going public until we're sure our infrastructure is able to scale, the data-loss-causing bugs are wiped out, and our feature set makes most people happy.

Look at the trends in electricity consumption per capita in California since the '70s versus the rest of the country. California has been about flat, the rest of the country has increased substantially. Energy efficiency is a hell of a lot cheaper than drilling or nuclear, and there's still a hell of a lot of low-hanging fruit.

You're missing the point. This is about demonstrating that it is a bad idea to use this technology for anything other than limited low-value transactions. The smartcard industry has been pushing "one card for everything" for years, and this shows that it's still a long way off. I'll keep my credit card and my travel pass separate, thanks.

I think his point about the iPod was it started out not turning the market around. See this sales graph from the Wikipeida iPod page (http://upload.wikimedia.org/wikipedia/en/8/80/Ipod_sales_200...). It was almost 2 years after launch until the iPod sales really started to shift upwards significantly.

I think his point is that persistence is important. It's not about apologising for being slow, it's understanding that you need a certain social capital to become big in a market. That can take time.

Agreed. The cost of implementing such an open system is negligible compared to the cost of securing it. You can harden a system as much as you want, but the business case for doing so is nil until it becomes a problem. At 0.000000003 of rides are being stolen it doesn't make sense.

While the security tzars are focused on the electronic hackers just having fun the majority of losses are probably coming from kids who jump the turn styles or go through 2 at a time.