Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module.

My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable.

So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk.

Unsurprisingly, in this configuration, it runs like the clappers.[1][2]

One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later.

[1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work.

[2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache.

The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB.

The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle?

I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.

It is not a bug, it is a feature. lighttpd used to wait 60s before checking the backend again. Nowadays the default is 1s. Set disable-time to 0 if you don't like it (it should be the default IMHO.)

http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModF...

It's that it got stuck in 500. I would have to log in every week or so and restart it when this happened.

You can also run a script to watch for PHP/APC segfaults and just restart the the cache.

I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

In my experience, stable APC releases have been very stable, but beta APC can poop out pretty bad, even if the changelog doesn't indicate anything that might affect your application.

I've used stable APC + PHP + Apache releases to do some large things, so not sure why APC is an utter bomb in 2012.

My gut feeling tells me that your profiling is correct, however - any significant PHP application (that isn't written terribly inefficiently) will have its bottleneck in data access and not the parse/compile stage. There's very little point to speeding up by 250% a portion of your app that only accounts for 2% of execution time!

Maybe.

... wat

> I cannot imagine running PHP without an opcode cache, you are losing a speedup of 300% to 500%

That's just not what my profiling showed.

There is most certainly a serious load reduction when using an opcode cache. It's not just common sense, you easily find a dozen independent benchmarks on the web proving it.

Now if you had the opcode cache running incorrectly you might not see the improvement (ie. some people misconfigure it where the memory is not shared and persistant, and instead gets destroyed as php children are created/removed).

In terms of microbenchmarks, opcode caches look spectacular. In terms of the larger stack, given the way Wordpress works? Meh, a few percent.

It might be worth it for a big site to shave a few dozen servers off the bill. But the overhead imposed by flakiness is not worth my time.

Problem is - you can't flush the output buffer.

Also, in that example, you can't use gzip. If I use a nginx+apache setup, I can do output flushing and gzip. I hope nginx fixes this issue.

*This affects php/fcgi.

Edit 2: Actually I think WP Supercache is the one that changed.

Edit 3: This config looks more up-to-date than mine -- http://rtcamp.com/tutorials/nginx-wordpressmultisite-subdoma...

I particularly liked how they used the try_file directive to cut down on the if statements.

----

Like most people I've cobbled together what I found through Google searching. My case is further complicated because I run a multisite configuration and use the WP Domain Mapping plugin.

It works in a few stages.

WP Supercache is configured to gzip pages to disk. It used to be that you needed to add extra code by hand to support domain mapping, but WP Supercache now cooperates and puts on-disk cached files for each blog in its own directory.

Then I put specific directives in a sites-available/ config file, not the main Nginx file (I serve a static site off the same server).

First,

    server {
        listen <ipaddress> default_server;

I like my logs to be divided by site, so:

        access_log  /var/www/log/wordpress/<sitename>.access.log;
        error_log   /var/www/log/wordpress/<sitename>.error.log;

        charset utf-8;

        root /var/www/wordpress;

        error_page 500 501 502 503 504 = /50x.html;
        location = /50x.html {
            root /var/www/wordpress;
        }

        location / {
            # Add trailing slash to */wp-admin requests.
            rewrite /wp-admin$ $scheme://$host$uri/ permanent;

            index  index.php;

            # Redirect /file/ URLs.
            rewrite    ^.*/files/(.*)    /wp-includes/ms-files.php?file=$1 last;

The next thing to do is try and serve a file straight off disk. Nginx does things in an unintuitive order, so even though this rule is further down, it will often fire first:

           # Rewrite URLs for WP-Super-Cache files
           # if the requested file exists, return it immediately
            # this covers the static files case
            if (-f $request_filename) {
                expires 15d;
                break;
            }

Now we proceed to determine whether or not we'll try to serve a cached file off disk:

            set $supercache_file '';
            set $supercache_uri $request_uri;

            # don't interfere with POST events (ie form submissions)
            if ($request_method = POST) {
                    set $supercache_uri '';
            }

            # Using pretty permalinks, so bypass the cache for any query string
            if ($query_string) {
                    set $supercache_uri '';
            }

            # Don't show cached version to logged-in users
            if ($http_cookie ~* "comment_author_|wordpress|wp-postpass_" ) {
                    set $supercache_uri '';
            }

            # if we haven't bypassed the cache, specify our supercache file
            if ($supercache_uri ~ ^(.+)$) {
                    set $supercache_file /wp-content/cache/supercache/$http_host/$1index.html;
            }

            # only rewrite to the supercache file if it actually exists
            if (-f $document_root$supercache_file) {
                    rewrite ^(.*)$ $supercache_file break;
            }

          # all other requests go to Wordpress
          if (!-e $request_filename) {
              rewrite ^.+?(/wp-.*) $1 last;
              rewrite ^.+?(/.*\.php)$ $1 last;
              rewrite ^ /index.php last;
            }
    }

    location ~ .php$ {
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }

      include /etc/nginx/clean.conf;
    }

    # Global restrictions configuration file.
    # Designed to be included in any server {} block.</p>
    location = /favicon.ico {
      log_not_found off;
      access_log off;
    }

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    # Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
    # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
    location ~ /\. {
      deny all;
    }

    # Deny access to any files with a .php extension in the uploads directory
    # Works in sub-directory installs and also in multisite network
    # Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
    location ~* /(?:uploads|files)/.*\.php$ {
      deny all;
    }

    gzip_static on;

Here's a list of "pitfalls" that are commonly used and really shouldn't: http://wiki.nginx.org/Pitfalls

I notice that you're using a lot of "if" statements. I hope you've read this http://wiki.nginx.org/IfIsEvil. I understand that in this case it might be harder to find alternatives for the if-statement, haven't looked into what you need really deeply.

The rtCamp configuration I linked to earlier makes use of try_files and says he has support for both supercache and domain mapping, so I think I'll migrate to that.

I really think Nginx in still underestimated by large. Most article handles rewrite in nginx based on their Apache knowledge.

Nginx's try_files is magical. So does maps{..} section.

Using Nginx map, you can server static files in WordPress-multisite without PHP (much better than X-SendFile or X-Accelredirection). On large wordpress-multisite network, this can really increase Nginx's capacity by multifold.

See - http://rtcamp.com/tutorials/nginx-maps-wordpress-multisite-s...

On a side note - you're completely right - whoever use nginx as a reverse proxy should just stop doing so and transition to nginx and drop apache.

It's not that Apache isn't good - it's just that Nginx is just way better.

Some of my sites have sufficient write activity that using XtraDB (basically a slightly souped-up InnoDB) is a smarter option than MyISAM. For searching people can just use Google, it does a much better job than MySQL's inbuilt fulltext search.

I don't know if they've fixed it, but AFAIK tables with full text fields used to cause MyISAM joins to go to disk, even if the full text field isn't included in the query. As you can imagine, this sucks. Maybe that's been fixed.

I don't follow how you already do that. Do you mean the OS disk cache?

Wordpress super cache plugin generates .html files from the pages and saves it in the disk for the os to cache.

Varnish leaves it to the OS to decide what pages to keep in RAM and which to leave on disk; essentially to avoid the double-buffering problem and because the OS has a larger view of the total machine's requirements.

But with wp-supercache I already have that approximate architecture. Files are on-disk, Nginx selects them, the OS notices that some files are frequently accessed and silently caches them in RAM. Everyone wins.

And I don't need to add ESI directives to my themes to get Varnish to handle frequently-updated material correctly.

If I remember correctly we were using DNS round robin and haproxy -> apache -> mysql all on freebsd systems wow have things come a long ways since then also it's incredible the sustained growth of Wordpress after all this time. good memories... congrats Matt on all your success.

There is no way in heck it's realtime queries.

I can tell when authors are in the WP backend on the server just by looking at the server load because it's crippling.

The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem of loading hundreds of files for any kind of page render (even to just authenticate for ajax, etc.) and over a hundred db queries in many cases.

A cache-miss in WP is a terrible, terrible thing.

They are talking about using nginx as a load balancer. It just proxies all requests off to the application servers that actually prepare the results. This article is really just about the fact that nginx can efficiently service a very impressive number of connections at once.

Those backend servers will be using caches of course though.

All logged out users should be served a completely cached static page that bypasses PHP entirely, it's way faster than even an opcode cache and way less load.

WP-Super-Cache can serve static pages to logged out users and makes this rather easy to setup. It's practically a must for wordpress.

For static caching I use nginx with fastcgi_cache. This makes WP-Supercache obsolete. I bybass the cache for requests containing certain cookies, so that logged in users always receive fresh pages.

Another big Wordpress performance sink is locales. If you have a blog in a non-english language consider using a language-caching plugin. Wordpress uses a PHP gettext implementation that is quite slow. 20% speedup if you cache the generated locales.

But it's a shame to spend days or months on tuning Wordpress, what is essentially just a blog engine.

It's also great to have the flexibility of adding more servers in case of a huge spike, even if you are running on only one server. Like on Heroku, where you can just increase the number of dynos in realtime. Doing that without a central cache that every server can access can break your MySQL because each new server will have a cold cache.

Pages are cached on NGINX for not logged in user, and when a new content is published all and only the pages that contains that content are deleted from the cached and regenerated at the next request.

So does my self-hosted wp blog, for that matter. If you are using WP Super Cache correctly, you can deal with significant amounts of traffic. I've seen peaks of hundreds of simultaneous users on my blog, with a server load < 1. The server is a modest aws small instance.

But multi-author blogs with thousands of logged in users is a nightmare with wp.

I suspect 90%+ of wp blogs are in the mini-blog category though.

Every time an article is published, it causes the cache to be deleted for not only that article but related pages, which means all those pages have to be rendered again.

For one author, that can be managed. Many authors, the cache is constantly being defeated.

That's irrelevant. It's the pages view count that counts, not how many authors are in the same cms.

- how do you do this ?

If you are compiling via source then you can also run "make upgrade" after "make install" and it will do it for you.

They have like 300 servers and who knows how many cores.

What kind title is this? The target High Scalability reader must be presumed to be a complete fool.

So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per CPU.

That seems like quite a low number, I would presume that the load would be shared amongst less servers than this and the others would be used for replication/redundancy.

I think we can safely assume each datacenter holds a dedicated load-balancing server. Let's assume they have only one of these at each datacenter, it gives us 70000 / (4 * 12) = 1460 requests per second per CPU, which is in line with reliably handling over 10,000 request per second of live traffic to WordPress applications from a single server.

and

In April 2008 Automattic converted all WordPress.com load balancers from Pound to NGINX. , which points to the fact that they're only talking about load balancers and they have several of them.

This link implies that one NGINX load-balancing instance cannot handle 70k r/s.

70 000 / (4 * 12 * 12) = 121.52 request per CPU

From the article it seems more like they have ~100, maybe less, for the load balancers which comes out to 175 requests/second per cpu which is getting a little more reasonable.

The point is the title is useless. It tells you almost nothing. That blog is a joke.

But yeah the title is pretty much useless.

Peter Westwood gave a talk on WP.com's infrastructure in London in January; I've tried to find slides online but I can't, which is a shame because he went into quite a bit of detail about their nginx/HyperDB/memcached/mogileFS setup.

http://wordpress.tv/2011/08/31/barry-abrahamson-ask-barry-ab...

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...