Can someone shed some light onto the executives hated homepage that was discussed? Since kissmetrics is an analytics company, doing this kind of thing is their core competency. How could executives change it to a loser immediately? There must be more to the story or some kind of other side to the story.
Shooting from the hip here possible problems with the "overly optimized" homepage:
1. Long term brand damage? If people go to the site and see something not as "polished" or "professional" looking, the bounces won't leave with just a neutral impression but a straight up negative impression. Sometimes even without a conversion a landing page can sell a visitor on a company and lead to recommendations or return visits down the line.
2. Lower value conversions? Is it possible that people who would immediately sign up without a long sales page end up being less valuable? This is more easily tracked so I'm sure OP would've accounted for this but it's still hard to tell without a ton of data.
This is really important, I think, and overlooked.
A/B testing micro-optimizes an outcome you're searching for. It doesn't tell you the long term trajectory of customers, because that takes way too long to get feedback. An example: "dark patterns". Facebook is deeply damaged, in part by making choices from well-run A/B trials.
Worse, A/B testing searches out local maxima. Maybe you get locked into an approach that precludes you from making the changes that would really drive the visit ultimately.
Performance indicators and properly using quantitative information are really important-- and many executives aren't versed at this. But, conversely, you can't use A/B testing to decide who you are as a company and define your relationship with the customer.
I dunno, I remember going to a talk at SXSW in 2011 about how A/B testing helps you reach local maxima but you need more to reach the next “peak”. I believe the story about Google testing 40 shades of blue was also in common currency back then.
I don’t think any of this stuff is new or undiscovered, really.
Well, I don't think anyone anticipated the "dark patterns" stuff at the time-- that we could be training technology to be kinda-evil in a way that would have massive consequences-- national political, reputation, etc.
From working in adjacent companies and doing marketing I can guess what the objections were to that:
1. It specifically mentions another company -> this often feels like a jerk move or from a branding standpoint something that diminishes your product. There's another front page HN article right now that leads with this:
2. It mentions nothing from the actual product/features which makes it hard to learn from the tests.
3. Some honest diversity questions with choosing that stock art guy as the face of the company. A tech company I worked at found out that we could get leads about 15% cheaper off of Facebook ads if we targeted only men and excluded women from the audience. We made the deliberate choice to use the less optimized combined male + female audience because it felt so ethically wrong to do otherwise.
I can’t speak to this group specifically, just to say that being an exec in an industry doesn’t make you an expert in that industry. When I worked in ad tech, I met plenty of execs who couldn’t run a facebook campaign to save their life. If you’re a founder of an analytics company hiring your CRO, it’d be great to find someone with an analytics background, but more than likely they’re coming in having worked in tech, but not that specific vertical.
It's specifically that vertical. Their product is designed for growth hacking, if they discard the results it's like saying their own product doesn't work.
Shooting from the hip here possible problems with the "overly optimized" homepage:
1. Long term brand damage? If people go to the site and see something not as "polished" or "professional" looking, the bounces won't leave with just a neutral impression but a straight up negative impression. Sometimes even without a conversion a landing page can sell a visitor on a company and lead to recommendations or return visits down the line.
2. Lower value conversions? Is it possible that people who would immediately sign up without a long sales page end up being less valuable? This is more easily tracked so I'm sure OP would've accounted for this but it's still hard to tell without a ton of data.