I don't think it stretches anything. The opposite -- that different websites, with interfaces that all vary subtly in a billion different ways, with different traffic sources, with different market positions, with different primary outcome goals -- are able to be compared? I would need to see quite a bit of specific evidence to believe that.
I've worked with brilliant conversion optimizers, in a single vertical niche where we had > 30 websites, each doing between 200K-2M visits/month. We would have loved to test + copy, but every time we ran this experiment, we found huge differences.
One subtle nuance is that "generalizable" needs to be more specific than "this change increased conversions". The true goal of optimization is not the find the B that beats A; it's to find the global optimum.
The Conversion industry tends to similar language as the complementary/alternative medicine space. "I did this thing, it had this outcome. I believe this happened because XYZ. It worked for me, it will work for you!" (People telling stories which may or may not be true.)
That is also why clinical trials for interventions is so important.
> The true goal of optimization is not the find the B that beats A; it's to find the global optimum.
FWIW, that's the goal of global optimization, a particual type of optimization, and one that rarely anybody on the market cares about wrt. business. Local optimization, "B beats A", is more frequently used, as usually all the business cares/can afford to care about is to find a C that beats B, if their competitors found the B that beats A.
Just because there are a billion possibilities with a billion traffic sources, different marketing positions, and you can do clinical trials and doesn't mean that providing someone the answer they expected and the possibility to search for other things beats just searching for things... which was what they were doing on the previous page.
If it was such a brilliant idea, DoorDash & GrubHub should change their sites so that links from their search results only take you back to their search page.
I've worked with brilliant conversion optimizers, in a single vertical niche where we had > 30 websites, each doing between 200K-2M visits/month. We would have loved to test + copy, but every time we ran this experiment, we found huge differences.
One subtle nuance is that "generalizable" needs to be more specific than "this change increased conversions". The true goal of optimization is not the find the B that beats A; it's to find the global optimum.
The Conversion industry tends to similar language as the complementary/alternative medicine space. "I did this thing, it had this outcome. I believe this happened because XYZ. It worked for me, it will work for you!" (People telling stories which may or may not be true.)
That is also why clinical trials for interventions is so important.