edit: a few additional details - from a few years ago, alas http://highscalability.com/blog/2011/3/3/stack-overflow-arch...

This math is obviously an average, but even if you halve that number you are still very much above everything most of us web devs are ever going to build or even see.

Also the data size they have of a TB or so means they can easily ("cheaply" enough) shove it into RAM, using a database like VoltDB (horizontal scaling/sharding, which seems like it'd be a perfect fit for vote recording).

There's still fun stuff outside of large sites. I'm writing a network search engine (record every packet and index it all) and even for small $10M telco, it can easily be a TB a week in signaling, plus a few TB of audio (but we don't tend to record every audio stream).

You'd be surprised how many intermediaries a call will go through. Sometimes even calling a neighbor, on landlines, will hit a tiny VoIP company in the middle. It's fair to assume, in such cases, random techies can record all sorts of end user calls without any real oversight. But I'm unaware of anyone actually bothering with this, usually the signalling provides all you need to figure problems out.

The only routine call recording I'm aware of is for 911 calls, for obvious reasons.

Anyways, my only point was there's lots of data even outside top websites and you can still have fun writing cool algorithms and systems. E.g. one of my todos is to implement SSE-optimized indexes. Maybe only a few percent boost, but fun and fairly justifiable use of time:)

BTW - CPU usage on a SQL server is usually low - as long as indexes are in RAM, lookups should be CPU-light.

In any case, SQL considers the oldest record with equal importance as the newest, and so index sizes can (and likely will) eventually be larger than data in some cases (two-column relation-tables with 20m records for instance). The options are basically 1. shard, or 2. consider a different approach.

We use NoSQL to store certain types of info, just as SO does - Redis, Elasticsearch, and so on. The more data we store and the more we use it in new ways, the more types of non-relational DBs we use.

I don't think we'll ever have a situation where MySQL goes away entirely, but it stops being the singular solution pretty quickly as you scale up.

I don't know how many users you have now and how quick you grow, but you can buy a server with more than 10x as much RAM as you have now. You may not "be able to scale vertically forever", but if you hit that spot you reached at least 5x Stack Overflow level. Congrats!

> We use NoSQL to store certain types of info, just as SO does - Redis, Elasticsearch, and so on. The more data we store and the more we use it in new ways, the more types of non-relational DBs we use.

> I don't think we'll ever have a situation where MySQL goes away entirely, but it stops being the singular solution pretty quickly as you scale up.

Sure why not. Use what fits your needs. But I guess you used Redis and Elasticsearch for it's features, not because it scales better - for that, some more RAM would have done it.

Huh?

I assume you're not referring to 122 MB of RAM. What's the "m"?

122GB.

That's still an abnormal amount, but it's in AWS RDS, on a db.r3.4xlarge instance.

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

Here's a grab from bosun taken just now of web tier load on those 9 servers: http://i.imgur.com/RJeEiYV.png

And here's a view of overall traffic coming into the NYC data center (except WebSockets): http://i.imgur.com/LV30989.png

Nick Craver Stack Exchange Developer & SRE

Companies with fewer users could have scaling problems earlier if their load is very write-heavy, if they generate artificial load from things such as crawling, if they have stricter latency requirements...

Moreover, note that Redis server. Stack Exchange uses caching very agressively [1], it is not the relational database doing all the work.

All that being said, I don't disagree in general: most startups should stop worrying about how they will scale their DB and stick with simple solutions that work.

[1] http://meta.stackexchange.com/a/69172

For example in my company, we went with Mongo because as I was told: "that's the industry standard", "unlake relational databases it let us scale and handle our huge amount of data" and also "it is schemaless so we are no longer constrained by DBAs".

Side note, the schemaless part is a bit fake benefit, because the data still has a schema, but the schema is now part of your. It seems like it makes things a bit easier, but you'll pay it with interest because as you are making changes the application is becoming more complex. Also I know there are exceptions, but if your data is highly relational (like in my company) storing it in NoSQL will come back to you back with vengeance.

Back to the original topic, we have several databases, we split them by function. One of them is a database that is used for geolocation (a bit more than that, but roughly matching IP to zip, or latitude/longitude to zip). I should also add that we previously used Coherence to hold that data and we keep it in RAM within 32 instances of m3.large (on AWS). Recently we decided to migrate away from Coherence to Mongo and it is considered a success because we replaced 32 of those m3.large instances with just 3 x m3.large instances. Huge savings it seems.

In addition to those instances keeping the data we have an infrastructure developed to move data from our authoritative source (Postgres databases) to Coherence/Mongo instances. We use AMQP for that :) so we have nodes that are used solely as brokers (and of course we have 10 of them because a single one was crashing under the load) we have producers(that fetch data from Postgres and push as a message) we have consumers (that take the data and put to Mongo) and several other nodes to control all of that and of course that's also "industry standard".

Of course we need all of that because (and this was especially true with Coherence) if something happens all data is lost and we need to "bulk-load" it. That process takes several hours because AMQP was not designed for huge binary transfers like this because it is a messaging protocol. And because we can't afford to be offline for entire day when something like this happen we have multiple environments so we can cut over to a backup one. Mongo supposedly helps here a lot, because the data is also kept on disk.

It's kind of interesting to see how one decision made many years ago in the past can affect bunch of subsequent ones.

Anyway, I did POC out of curiosity. My reasoning was that if we treat Postgres as the authoritative source, why not keep it in a way that is directly query-able (currently the data is stored in a way that makes it easier to bulk load it).

The results were better than expected. Not only it is faster (with PostGIS and ip4r extensions every query takes fraction of millisecond to run), but also the data which was about 8GB of it, after loading it to Postgres it was only 500MB.

How is that possible? Well, because Mongo (and I suspect Coherence) has just simple indexing capability, in other to have a reasonable performance, developers decided that for every row that holds IP range they would simply list all IPs in that range as 64 bit integers and placed an index on that. I would love to see what would they think of if they would have to add support for IPv6 as well[1].

I'm currently having an uphill battle to convince other teams that this is the right path. I really would like to skip the crap and go with this higher, but that's probably unwise.

[1] 2^128 is more than estimated number of stars in the universe.

Because the data is stored in such inefficient manner in order to scale at one point they decided to separate it into different databases by type of the data, I only used geolocation as example.

StackExchange runs on physical hardware and they have spent a considerable amount of time optimizing for bare metal performance. Their team is unique in that they embraced hardware from the start vs. many teams today that want hardware abstracted away. They have more maintenance overhead around hardware management, but don't experience lost IOPs due to IaaS (AWS/Azure) overhead.

Their current environment required overcoming tremendous technical hurdles on earlier versions of SQL Server (these might be general RDBMS limitations as well). Luckily they were able to get the Microsoft SQL team engaged to get them through this.

Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and others) are highly respected members of the SQL & .NET community.

It's easy to look at their setup and say "Wow, that's not a lot." and overlook the circumstances & talent required to achieve such an efficient system. I'm not sure many teams would pursue this architecture if they knew the effort (and luck) involved.

Jeff Atwood's goal was always "performance is a feature," and that means rendering your pages faster than anybody else's. With that in mind, you're always going to have technical hurdles to overcome on any platform - because you want to get faster results than anybody else is getting.

I remember when I first got involved, and Jeff told me something along the lines of, "This slow query runs in ~500ms, and we want it to run in ~50." That statement alone is a huge leap over what a lot of RDBMS users say - usually when people refer to a slow query, their unit of measurement is whole seconds, not milliseconds. They were serious about performance from the get-go.

> Luckily they were able to get the Microsoft SQL team engaged to get them through this.

Nothing against Microsoft - I love 'em dearly and make a great living with their tools - but Stack's success is much more due to their own internal team's dedication to performance. When we open bug requests with Microsoft, the time to resolution is typically measured in weeks or months. During that time, Stack Exchange's team has to come up with creative, impressive workarounds. They're the sharpest SQL Server users I know.

> Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and others) are highly respected members of the SQL & .NET community.

Awww, shucks, but I'm not brown-nosing Stack when I say that their current team is ridiculously good. Their Site Reliability Engineers know more about SQL Server than most career DBAs I know.

> I'm not sure many teams would pursue this architecture if they knew the effort (and luck) involved.

It sounds like you're implying that other architectures are faster by default and with less effort, and I would disagree there. I haven't seen any big sites (big being top-100-in-the-world[1] type stuff) where the persistence layer was set-it-and-forget-it. Scale - especially millisecond performance at scale - is seriously hard work on all platforms.

[1] https://www.quantcast.com/top-sites

> Luckily they were able to get the Microsoft SQL team engaged to get them through this.

Looking at this statement now, I see that it might have appeared that Microsoft 'came to the Stack team's rescue.' My impression was that over time Microsoft alleviated some of the workarounds the SO team was running into.

> I'm not sure many teams would pursue this architecture if they knew the effort (and luck) involved.

I was simply stating that the SO architectural graphic looks deceptively simple and I can only imagine the amount of drama on hardware alone the team went through to achieve their goals. I do believe there are other OSS-based architectures (with likely more layers) that would require less 'workaround' effort to deliver reasonable performance/reliability, but compounded with SO's likely SLA & perf requirements on a closed-source RDBMS? It wouldn't surprise me to see some go 'good enough' and move on. I doubt anything is Ron Popeil easy when your trying to shed milliseconds on a persistence layer.

Hope this clears things up a bit.

Unless we're testing beta software, the MS stuff generally works and works very well. We of course work with them on feature requests that will make life better - some happen, some don't. I'm on the opinion you should try and have this relationship with any vendor or open source project. We're trying to make what we use better, and not for just us.

Nick Craver - Stack Exchange SRE

That said, there are so many benefits of running multiple servers behind a firewall off the bat, I can't fathom why people don't do this. It more or less enables trivial failover for many classes of problem, and deploys are no longer as large a source of instability: deploy by launching a new box, route a small amount of traffic to it at first, then route all of it. If there's an issue, route to the previous app servers.

Compared to the time wasted by getting the architecture wrong off the bat (and months of pain while you wait for there to be enough time to do the work, which never happens), this is a massive win.

But the specifics are heavily dependent on the actual code executed; to a first approximation, most Rails code is shuffling parameters to and from a database. It's very difficult to speculate. If most of your Ruby code, to take an example, is effectively a wrapper around native code (e.g. regular expressions or string substitution in templates), then the win from using something like C# or Java is probably low.

I think the complexity costs of scaling out are underestimated; or rather, they're just accepted as a fact of the matter, like there isn't an alternative. More moving parts in the name of redundancy etc. usually makes things more fragile, without very careful design and implementation.

Interesting that you bring up GitHub. GitHub was probably the most unreliable service our startup used before we moved to Stash. You could count on it being down for a few hours every month, reason given usually "DDoS". We moved to Stash for contractual reasons relating to security and ISO certification required by our target sector, finance, but I could see us having done it for reliability too.

If they were using a slower language, they'd get much higher latencies (maybe 200-300ms instead of 33ms) that couldn't be fixed no matter how many servers they add. If you read the linked page and Nick Craver's blog carefully, you'll notice that they're deliberately aiming for very low latencies everywhere.

As with all things, you have to measure it for your own application and find your bottlenecks before drawing conclusions.

I suspect if asp.net MVC hadn't been out, it would have been written in webforms.

I seem to remember asp.net MVC had only been out a couple of months when they started writing it or not even fully released, I was learning django at the time because MVC so was so obviously better than webforms. But because c# was so much faster than either ruby or python, when asp.net MVC turned up, and it was very, very good for a v1, I stuck with c#, as did many other .net programmers I suspect.

At the time SO was written rails and django were still new and slowwwwww, php was king and still mainly written by page without frameworks, Java who knows, and the MVC revolution was just starting.

I'm not sure what you think he would have written it in? I have a feeling if you hunt through coding horror you'll probably find a post from the SO start time period saying "use what you know". In Jeff Atwood's case that's C#. I know that Joel's company had written their own language that I think at the time was still in use, so they wouldn't use the fog creek stack. Which was still based on Microsoft stuff anyway.

That said, SO was indeed a complete game changer in terms of the rapidity with which one could get solid information.

It might have been easy to go php > ruby, but C# > ruby meant massive amounts of pain or complety changing your environments. While my company at the time had VMs, they were all windows and it wasn't a case of simply spinning up an EC2 to play. The pain of leaving a MS stack was massive.

I'm using ruby/python a bit interchangeably as I was.playing with both, but even things like gem took a load of effot compared to now. Like a couple of days of fiddling doing nothing interesting, not programming but constantly hitting errors or tutorials not working as you'd got a step earlier in the chain slightly wrong. It was a huge barrier to entry.

The worst tutorials claimed to be for both windows and linux, but would link to an out of date windows installer and all the command line commands would be for linux. I once made the mistake of thinking I'd write little maintenance scripts in Python instead of VB, it was great until I tried to connect to a MSSQL environment, ended up in a world of pain.

Last time I span up node.js on a new windows computer it took me like 10 minutes. Everything has changed.

That said, I see what you're getting at now and it isn't what I thought you were talking about previously. It looks like what you're talking about is it taking a few days (or so) to get a working environment up and running vs. nearly instantaneous now? If so, fair enough I totally agree.

I thought you were talking more about the more nuts & bolts parts, like it taking a few weeks to come up to speed on Python (for instance) instead of several months. I've always felt that claims of it being difficult to change languages or other type things (e.g. changing from Oracle to MySQL) were radically overblown, and these days w/ sites like SO it's pretty much trivial.

These days the landscape is a lot better, especially when it comes to Ruby.

You're correct that ASP.NET MVC was new at the time, but with guys like Scott Hanselmann comparing it to an acoustic guitar production versus WebForms more full-studio approach, and with Phil Haack on the squad, and with the corporate blessing from Microsoft and promises of future integrations, this was not some fly-by-night project that required a lot of faith to invest in. It was a clear case where Microsoft got something right, thanks to a small team of talented devs.

Consider: for the first time in history, you had the ability to write clean C# code in a web context with full control of markup and proper separation of concerns, sans cruft or baggage imposed by attempting to stretch a "desktop" metaphor over an request/response medium. Don't underestimate the psychological effect of this on programmers who'd been spending their days in the trenches builting line-of-business CRUD apps with ASP.NET WebForms and unbridled VIEWSTATE. So to me, ASP.NET MVC was the clear and obvious choice for StackOverflow. If you accept that, what better language than the C# he already knew?

Beside that, what matters here is not throughput but user-visible latency. And here's where a fast language shines.

If they are capable of rendering any question page in 33ms, it stands to reason they could use just 3 web servers and still serve content in roughly 100ms (which is fast enough that most people wouldn't notice).

My point is, without knowing what is the bottleneck for the web servers (we only know it's not CPU - I suspect it should be I/O) it's hard to make this kind of point.

Aside from that, I think I remember the original development team was already familiar with C#/.NET, and this drove the decision for the original stack.

It doesn't work that way. With less servers, requests would still take 33ms (until the servers are overloaded, at which point they may start to take longer unless they have rate-limiting to reject requests above their capacity), but their overall capacity would be lower.

You don't need to be CPU-bound before this starts happening because requests don't arrive in evenly spaced intervals. Decreasing CPU load, even on a moderately loaded, I/O bound system, can still reduce latency.

We know from the provided data that their architecture can compile and serve a page in about 33ms. This includes everything that is not web servers, so we are certain that the actual time spent in the webserver will be lower.

Aside from that, we know that their webservers are nowhere near full CPU usage (they claim 15-20%). This also goes in the direction of not invalidating the use of 33ms as an upper bound. We would need to know more about why they have 9 webservers in the first place to get a better idea of what's going on (mostly how the I/O is behaving, I'd say).

Putting this assumptions together, we can safely assume that by shrinking the number of webservers from 9 to 3 wouldn't cause the page loading time to increase to more than 100ms (and it's likely going to be much smaller in most conditions).

Of course, this may be overlooking some piece of information we may be missing (the prime example still being I/O-related).

edit: I see now that in the original post I stated "roughly" 100ms, while it would have been better to say "at maximum".

It's not about optimization at all.

I'd be interested to know what DBMS they're running, and it's very interesting that they've moved the tag "indices" out to their own cluster. I'd also be interested to know whether they run out of 1 datacentre and how/if they mitigate that risk.

update: http://www.quora.com/What-will-Quora-do-with-the-50-million-...

"We project a large portion of this money to go to EC2 and other AWS bills. It might be replaced by whatever the most appropriate place for us to run our infrastructure is in the future but as of today it's looking like EC2."

MS SQL Server, latest version probably

The Tag Engine is entirely self-contained, which means not having to depend on an external service for very, very core functionality. It's a huge in-memory struct array structure that is optimized for SO use cases and precomputed results for heavily hit combinations. It's a simple windows service running on a few boxes working in a redundant team. CPU is about 2-5% almost always. Three boxes are not needed for load, just redundancy. If all those do fail at once, the local web servers will load the tag engine in memory and keep on going.

Apparently, it got leaked :) http://meta.stackexchange.com/a/244932

We kinda wanted to have a 2.0 version of the page with live data before promoting it. This one is static with #’s we sampled at the time.

They would possibly have to be in separate datacenters though, since shipping large amounts of data over seas can be inefficient and possibly expensive.

I'm not sure at what point that starts to be a necessity, they're big but it's still a niche market. And their content is very sparsely visited (i.e. it wouldn't happen often that 80% of their audience would view a single question).

Storage requirements would change of course, but bandwidth would be absorbed by whatever CDN they're using, so traffic quantity stat would be different.

Also, with images/videos, they're now sending more URLs/embeds/etc. along with data, so more characters = more RAM for caching and/or serving up on web servers. In theory, shouldn't be much more of a hit.

They started with BizSpark, Microsoft's program for startups that gives you free licensing for a few years. After that, you pay maintenance on what you've got, and buy new things as you add new servers.

For production? No. Development machines? They used to.

The best you can hope for these days as a showcase ISV/startup is more direct access to devs, and you'll probably only get that if you are proofing a new system of theirs. Remember, the most likely gospel is that Microsoft is expensive to run - so dropping the licensing cost "for PR" is actually a bad idea.

> For production? No. Development machines? They used to.

Development licensing for SQL Server (and most of MS's products) is done through the MSDN program, and it's dirt cheap, and doesn't require any leeway. Anybody can buy MSDN licensing.

For production, everything is negotiable. It's not unusual to get special SQL Server licensing deals when you buy in bulk.

(That's me walking the line between what I can say publicly, and what I can't.)

There are other solutions to take it much further than that.