If I was to use Varnish, would that make my need for Redis obsolete - should I switch to a non-memory based key-value db and use Varnish on top of that?
My current stack: nginx/php-fpm/redis. Nginx and Redis serve me well, but php-fpm makes my website rather slow with high volumes of traffic, so I believe the solution for that would be Varnish(?).
so @ tumblr, we're using varnish for a full page cache (we use it for parts of the API as well for response caching), and invalidate when a blog updates (or your page can just TTL out).
I definitely agree that I wouldn't use Redis (or memcache for that matter) for storing entire pages and should be used for more of an object cache. Even then, we use memcache for "simple" data structures and when we need more complex data structures, will use Redis.
Redis is great if you need some kind of persistence as well (and it's fairly tunable), where as memcache and varnish, are completely in memory (varnish 4.0 I believe is introducing persistence). So you kick the process, and that's all she wrote for your cache until it gets warm again. (Which has its own challenges).
Varnish also gives you a language called VCL to play around with to maximize your caching strategies and optimize the way varnish should be storing things. It's got an easy API to purge content, when you need to purge it and it should support compression for your pages out of the box without too much tuning.
If you're having issues just speeding up static content, give varnish a whirl. Spend some time with it, and you won't be disappointed.
I believe you can also look into using nginx as a caching alternative to cache responses, but I don't have too much experience with that. I've heard it used with some success though.
No. Redis caches single objets, Varnish caches whole pages (unless you're using Redis as a full page cache).
If you're using Redis as your database, I'd suggest not doing that anyway, as you'll start running into problems as your dataset gets bigger than available memory and it has to start swapping. I've found it works much better if you use it like memcache with a richer set of data structures.
Tough to quantify "obsolete." You can put caching layer upon caching layer upon caching layer, but diminishing returns kick in almost immediately.
Ideally you have one caching model to rule them all, unless you're doing a lot of module-specific caching (ie, this element should be cached longer than this element, etc.)
Well at the moment, I would consider Redis more useful than Varnish due to expired events. As to follow the "one caching model to rule them all" rule, how about using ngx_lua to dynamically change a static .html file with the data from Redis thus cutting out php-fpm (unless they are logged in, then requests will go directly to php-fpm)?
Or use Varnish, use Redis for expired events but with an empty value and use keyspace notifications to automatically remove the data related to that key from the database and purge Varnish.
My current stack: nginx/php-fpm/redis. Nginx and Redis serve me well, but php-fpm makes my website rather slow with high volumes of traffic, so I believe the solution for that would be Varnish(?).