I'm glad they are doing this because they need to do something about memory usage, but there are some problems with this approach.
-websites change over time. If a site gets redesigned, it'll throw off test results unless they're rerunning the tests on old builds too
-this still doesn't capture how users actually use browsers so could miss some serious leaks. What about clicking around on the same site, opening links from the same site in multiple tabs, leaving a tab open for a week, etc?
Additionally, they keep saying that extensions are a huge source of leaks. I feel their pain because they don't have that much control over them, but they need to come up with some sort of a solution to deal with it, because let's face it, who's going to run FFx without any extensions?
Their FAQ [1] states that they were fetched last year and stored locally for testing. So changing sites shouldn't affect the result. I presume they will switch the set of sites in a few years, statistics don't need to be over 10 years to be useful.
Hm, they stated that the gaps in the graphs were due to failures collecting the data for that release. To me that implied that they weren't rerunning the test on each version every time.
Extensions should be given an aggressive memory limitation to work with, and a profiler should be made so that extension developers can see what they're using and where it's spent.
Even if a per-extension memory limit was possible, good luck choosing the actual limit and then surviving the storm of "WTF did Firefox kill my extension?" complaints.
For various reasons this cannot be done for most traditional extensions that introduce advanced functionality (e.g. AdBlock Plus), since they can override the browser's functionality without limitation. Add-on SDK extensions are another matter but those are comparatively very limited in what they can do.
This can be fixed with a split model. Chromium sometimes introduces fairly ad-hoc APIs (there was one to hook into HTTP requests, first declaratively which was discontinued, then with JS-side filtering which seems to be used by the ABP port); Jetpack add-ons can also use custom modules that hook into the full Firefox API. That way, the part that needs care and auditing for memory/lifetime issues can remain small.
Additionally, they keep saying that extensions are a huge source of leaks. I feel their pain because they don't have that much control over them, but they need to come up with some sort of a solution to deal with it, because let's face it, who's going to run FFx without any extensions?