I really don't see Flickr's (or Forrst's) approach a "trick" or feature. I think most deployment techniques come out of a desire NOT to deploy like Flickr(or Forrst).
Right now, yes. Deploys happen frequently enough and are sufficiently fast that I can get something disabled globally in under 30 seconds. The next iteration of this stuff I'm building now; goals are standalone library, more DRY, and some kind of simple web UI where I can do the global enable/disable actions. I'll likely stick to manipulating buckets in config files for the foreseeable future; it's being able to mass-enable/disable that I need to refine.
I've started using Redis key/set operations for things like this at GitHub. The nice thing is that these techniques are all simple, and it's easy to modify to fit your evolving needs.
Yeah, I definitely considered using Redis too; I'd be able to leverage web UI for even stuff like adding/removing users from buckets. How are you handling with persistence? I presume backing up the dump.rdb file frequently? I'd be a shame to lose any of the data (though I'm probably being overly paranoid)
Using a combo of the append-only file[1] and a redis slave[2] you'll get about the best durability one can ask for in today's world (assuming you're aware of the gotchas with regard to expiring keys — an edge case).
We were already using Redis and had an existing backup plan. Though if you're using Redis solely for feature rollout, you can likely get away with it failing gracefully in case the server is down.
Feature flags are essentially identical to operating A/B tests in this respect, right? I have a magic button which logs me in as a target user, and it reproduces exactly what they would see, including the results of A/B tests. This helps to reproduce unanticipated interactions between tests. (This has only happened maybe twice, but they were doozies.)
Not really. You just add extra tests for your new functionality as usual. If you're replacing old functionality, you'll have duplicated tests until the feature is rolled out for everyone.
I think the key is being able to tell if bug reports are related to an activated feature. For instance, our error tracker tells us which user account triggered the error. From there, we can see what new features they have enabled, and factor that into our debugging.
