In it, Dustin Curtis criticizes Google (rather rightly, in my hindsight-enhanced opinion) for a Marissa-Mayeresque design process of "Make 300 logo variants with pseudorandom permutations, and then pick your favorite." Looking at those permutations now (http://2.bp.blogspot.com/_7ZYqYi4xigk/SEnK37orPGI/AAAAAAAAAp...) it's obvious (again, with the benefit of hindsight) that they were all very inside-the-box. Apparently nobody even considered trying a different font.
The reason they did that was so that they could build up data on the effect of each factor independently. That way, when the bigwigs get into a meeting and discuss alternative designs, they can predict "Ok, this will have $X effect on revenue but will decrease search latency by Y ms, which itself will have +$X effect on revenue."
I was the first engineer on the first visual redesign that tried to change everything about the page [1]. One of our biggest problems was that since we tried to change everything at once, the metrics went haywire, and we couldn't know if a change was because we introduced a left-nav (okay, actually we kinda could, because we'd experimented with a left-nav on its own beforehand) or because we changed the line spacing or because we cleaned up the logo or because we added icons. There was effectively no way of "debugging" the user; when user behavior varied from expected, we didn't know why.
I've got plenty of other stories from that era, but they aren't really fit for a public forum. Happy to discuss privately. (Kinda ironic when the non-Googler says that to the Googler...)
I'm surprised no one on the team was familiar with multivariate analysis. That's the wrong way to say it. Let me rephrase more carefully...
I'm surprised no one used more sophisticated analysis techniques to allow an experiment that changes many explanatory factors simultaneously and estimates the effects of not only each factor independently but also their interactions. It's relatively straightforward statistics and would have been taught to clever undergrads or first-year grad students who specialized in math, economics, machine learning, industrial engineering, etc.
There were stats Ph.Ds on the team that were certainly familiar with it, and I knew at least one former Amazonian manager that held that we were too afraid of having multiple experiments running at once and too afraid of confounding factors. (Amazon's A/B testing is much more traditional split-testing, where they show one variant to half the population and one to another half and then pick the best one.)
The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality.
Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations."
tl;dr: Largest and best-funded social science program I've ever seen.
> tl;dr: Largest and best-funded social science program I've ever seen.
I don't want to sound like a hater and I'm very well aware of the fact that some design choices might have affected the company's revenues, but all this discussion involving very smart people discussing what, from a very "outsider-ish" perspective, seem like trivial issues (the color of the letter G, for example) reminds me of Herman Hesse's "Glass Bead Game". (https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot)
>Largest and best-funded social science program I've ever seen.
You should check out Facebook. There emotional manipulation studies are horrifying, but they certainly have access to many more person hours of time and collected data than Google does on end users.
I understand the urge to try to maximize on stuff like this, but in the end it loses sight of what's most important: the quality of the results. And sometimes I feel that a lot of valuable engineering time and effort is wasted on making things prettier and better monetizing in the short term when in the longer term the better search results are what would really move the needle.
And most A/B testing focuses on the very short time effects of the change (and change all by itself, even change for the worse can have a short term positive effect just because something is new). On another note: A/B testing has another limitation, which is that you're optimizing for the bulk, please one person more and you more or less automatically annoy another.
There are plenty of talented people working on ranking and webspam too.
There's a limit to how parallelizable those problems are. At some point, more people working on an algorithm doesn't make it faster or better, it just makes it more confusing and worse. Mythical Man Month applies as much to Google as it did to IBM. And so you might as well put people onto optimizing the rest of the page as well, because they'll do more good there than having yet another cook in the ranking algorithm kitchen. (I'd argue that there are too many people working at Google in general...I left, so I'm no longer contributing to that problem. But then, there were probably too many people when I joined, so I contributed to it for 5 years.)
The worst limitation of A/B testing is that it's not producing anything better than a local maxima. Nature does indeed work like that, but it took millions of years along with random mutations and we don't have that much time. And if in our thinking process we would have relied on optimization algorithms favoring the local maxima, then Jazz wouldn't have been produced, amongst others.
Can you offer any perspective on the whole sometimes-named "mystery meat" UI that's been rolled particularly but not only into mobile design over the past half decade?
Further, more and more of this seems to have been "backported" into traditional desktop presentation design.
While I will experiment around until I more or less "get it" (although I may miss finding some useful features for months), it particularly frustrates people like my parents. Mom is somewhat spatially challenged (although plenty bright in other ways), and I've had numerous calls with her where the controls for what she wants to accomplish have changed and simply bewilder her. AND, given her spatial orientation problems, sometimes we will have to have the same conversation several times -- because the rather arbitrary-appearing icons and patterns don't immediately "sink in".
P.S. I mean this as a legitimate question, if it fits the parent's -- or others' -- field of experience. It seems that a lot of this design is expected to simply permeate and be picked up by osmosis. Is that really the case?
I'm curious what you mean by "mystery meat"? I assume it's the use of icons and drawers to trigger functionality rather than labels and links?
That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media. I've seen a bunch of mobile apps with help overlays or tour videos the first time you run them, which can help a lot. It gets back-ported to desktop apps because once people do learn the functionality, it leads to a cleaner, less cluttered UI.
I'll also point out that early PCs had the same problem. A typical PC from the early 1980s had 320x200 resolution, a little less than a smartwatch does today. For most of the 80s and early 90s, the standard was VGA (640x480), slightly smaller than most smartphones today. Icons were really common back then as well, they weren't all that discoverable, but at least people read manuals occasionally. Most folks don't bother now. Menus were another response to that problem, but they don't really work on mobile because the tap target is too small to efficiently hit it (the navigation drawer & hamburger menu is the modern-day equivalent of the menu).
The web - with its inherent discoverability and all of its space for explanatory text - didn't really become popular until 1280x1024 displays became widespread. I remember designing webapps for 800x600 screens; it was painful, and they had discoverability problems too.
Well, a, um, "literal" example was the initial occurrence of what I've just learned is called the "hamburger" icon/menu. Which was "backported" to Google's desktop designs.
A particular instance I recall was changes to the Calendar web page presentation. Mom initially struggled a fair amount with the controls in the left bar for selecting which calendars were displayed. This was in a Google Apps environment with 20-ish people participating.
I remember when screen presentations could be somewhat cryptic -- per your description as well as other instances. But, as you point out, there was usually a manual. Now? Often it seems to be down to guessing and poking around. Or asking friends. But if you are a bit older and not constantly talking over such things with your friends, that means of edification is somewhat starved.
P.S. Thanks for the response. And as I said, I'm curious about the perspective of someone on "the other side". I'm not trying to be unduly critical, here; just, this is an issue I've consistently bumped up against, from my own perspective.
> That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media.
Some of the most effective solutions to this I've seen require solutions beyond visual representation. They address the problem through dimensions independent of viewport size and space.
For example, LukeW's "Don't Divert the Train" http://www.lukew.com/ff/entry.asp?1798 moves content from behind navigation and into the user's already-occurring action stream. The button is removed entirely, as is the designer's need to consider how communicative icons are for representing actions.
This requires a broader view of how features relate to each other through a design process that discovers other paths to successful behavior. Approaches like the hamburger menu, in addition to the usability issues you note, solve functional goals but have fallen out of vogue for their ineffectiveness at encouraging specific actions. Label-less icons, while attractive and convenient, suffer similar problems. As getting users successful quickly is generally an important goal (before they've internalized your app's unique visual language), the current trend of putting core functionality behind opaque symbols seems counterproductive.
Another thing I'm now reminded of is, on Android, the... functionalization/clickification of text.
Some text is active, other is not. I don't see any visual cues that distinguish the two. I end up just poking at text, or thinking, "If I were designing this, I'd want more detail or a link to further controls." In the latter case, sometimes I find I was thinking the way the designer/programmer was, and that text does indeed lead somewhere.
For someone who doesn't "think that way", this is more of a challenge.
This also seems to have become more prevalent in desktop / full screen design. Stuff that doesn't look "clicky" is.
I think that was one of my mother's problems with changes to the Google Calendar web page UI.
I worked at a major tech company where every decision needed to be justified by AB testing. Our design was a series of haphazard guesses made on data that depended on a very brittle data pipeline. Needless to say, nobody thought it was a very good product, but the incredible bureaucracy and people behind the approach made it hard to fight against.
It shouldn't be a problem. In science, everything must be tested and proved if you want to know the truth or the effectiveness of an theory. Design is a science, so it can be measured and developed by tests of all kinds, even emotionals.
Whereas a design studio with sufficient resources should fare better with a data-driven approach, a sole designer or small team with time and feedback constraints will probably do better with traditional methodology. Traditional methodologies might have certain quack dogmatic rules and processes, but it offers a battle-tested baseline.
> One of our biggest problems was that since we tried to change everything at once, the metrics went haywire
Isn't that the problem with empirical methods in general? They are great for measuring the effect of small simple things (like small changes to a web page), but are much less effective in measuring bigger things (like say, programming language A vs. programming language B).
Absolutely. Business is an interesting field because you derive a fairly large advantage from being correct about reality, but that advantage counts for nothing if you're so slow that somebody else changes the reality while you're figuring it out.
I'm not sure how it works at Google, but adding a random query string usually helps because of how proxy servers handle caching. Proxy servers usually treat an URL with different query string as different resource and cache them separately (e.g. /posts?page=2 is definitely not the same as /posts?page=1).
This is not only applies to caching by ISP, but also includes caching by the website itself as well (some websites put proxy in front of their server to cache rarely changed portion of the site, or even using CDN.).
Anything after "?" is a query parameter that the server can read and then apply additional logic to for a response. In this case, they have code that responds to new logo and returns the new logo instead of the previous one.
It's actually not querying anything, since it works for any query you pass. What happens is that you're bypassing cache because you're loading a different page, as explained by sirn above.
But all three logos were designed and introduced at the same time as their corresponding favicon. Your claim was that the favicon was just a copy of an existing part of the logo.
Your claim was that a favicon could be a different font than in the logo. But these examples are all a part of the logos. Google would have needed to rebrand their logo to do the same.
I think you're mistaking their symbol for "a different font". Each one of those has a logotype and a symbol, which make up their visual identity (the word 'logo' is used interchangeably for those, or their combination). They are merely using the existing symbol as favicon - the stylized N/B/B. Google's symbol at the time was the uppercase G in the same font, so creating a new one for the favicon would be considered a rebranding.
But it's just an image right? Does the browser decide to treat the image different because of the format or the extension? I'm pretty sure you can use png files as favicons if you want.
Anyways I tried it in IE, including restarting it and it still showed the old lowercase g.
I've never understood that criticism. Business decisions should be data driven. You test new signup flows to see which one onboards users better. You test advertising changes to see which one converts better. Why should a logo design or a title bar color be any different? Without testing each iteration, how do you know which one has the best effect on whatever success metrics you're using? I can understand if you're worried about optimizing to a local maxima, but that's not a criticism of A/B testing, it's a criticism of the diversity of the available inputs.
User testing is great, and very useful for so many people.
BUT, when you're the size of Google, or Facebook, or Microsoft, Things look a bit different.
Most people are far more conservative than they realise, but also, perversely, adapt quite quickly to change if they experience it day-to-day. This means that if an interface has enough inertia, then a change that performs badly in user testing, can actually quite quickly be accepted by people. (PROVIDED it's actually better).
In the first example, every change to the Facebook UI is met by a barrage of criticism and hatred, but they seem to actually make good decisions, so the changes get absorbed by users, and things continue.
Having said that, Microsoft struggle, because they often make changes that are fundamentally bad decisions, and have to back-down. For example, the Office ribbon took a while to be accepted, but is /generally/ liked nowadays by people, because it was functionally a good change (yes, some people will disagree, but the majority wins on this front). Taking broad user testing results on the ribbon would probably have stopped microsoft from progressing with it. On the other hand, the 'Metro'/Tile UI in windows 8 was an actual usability disaster, and has forced them to revert to a more common pattern until someone can design new metaphors that actually improve usability in windows.
Finding test subjects that aren't used to, and comfortable with, the google font/design styles, to give objective feedback on a change, is really hard.
[ I didn't mention first time round, but I /really/ don't like the new logo, that lowercase g makes me cringe inside every time I see it. ]
The evolutionary, data-driven approach will never yield a Nest thermostat, iPhone, Helvetica, or flag of New Mexico. We have no idea how to even begin to write a fitness function for good design.
I'm not sure where you think taste comes from besides evolution of ideas and data analysis done by black-box functions inside people's heads.
Formalizing that process and using externally-collected data is just de-black-boxing something that happens in a black-box form in the professional's mind.
Also, making decisions based on wishy-washy difficult-to-quantify human traits like "genius" and "sense of taste" seems unnecessarily risky, compared to making them based on measurements from repeatable tests.
I'm not advocating design by committee. I'm advocating justification with data. The designers should propose candidate designs. But the decision about which one to go with (or whether to redesign at all) should be supported by whether going with it furthers the business's goals.
You point out the need to pick the right measurement, and I agree, but it could be as simple as "What's the effect on conversion?" It has to be something! You can't just say "We're re-designing because I'm a HMI genius."
When a company is capable of replicating those black-box functions as an algorithm or business process, then it will be ready to launch the era of automated design.
Until then, the best known way to design something nice is to hire a talented designer and trust their instincts.
> The evolutionary, data-driven approach will never yield a Nest thermostat, iPhone, Helvetica, or flag of New Mexico
...for now, given today's technology, but I wouldn't bet on it long term. History is full of examples of things we once thought only humans could do.
But once many candidate designs are available to choose from, what better a way is there to unambiguously decide on the best besides testing and measuring?
As I said, the problem is writing the tests and knowing what to measure. How do you discern between a bunch of tests and measurements that are poorly-designed, and a bunch that are well-designed? You gather a bunch of people with good taste, and trust their gut.
But at that point, why not just cut out the middleman, forget the tests, and let the artists be artists?
signup flows to see which one onboards users better.
The trick is: there is no global "better" — only 'better', given a specific group of users.
Which users though? Running a million 40 year olds from Iowa through your website flow will generate a widely different result from running a million 13 year olds in Atlanta through the same process.
Google is still stuck in the "one design for everybody in the world" mindset, so they have to be appealing to everybody, which naturally makes them bland.
success metrics you're using
Success metrics! The paradox of these hyper-optimization methods is they largely apply in hyper-trivial situations. Once you are big enough, you just tell people how things will be. They don't get to have a say anymore.
And you can't rely on past data to create new trends because that data doesn't even exist.
Data said that children's books don't sell. But then Harry Potter happened. Data also said that people loved phones with keyboards. Then the iPhone happened.
In it, Dustin Curtis criticizes Google (rather rightly, in my hindsight-enhanced opinion) for a Marissa-Mayeresque design process of "Make 300 logo variants with pseudorandom permutations, and then pick your favorite." Looking at those permutations now (http://2.bp.blogspot.com/_7ZYqYi4xigk/SEnK37orPGI/AAAAAAAAAp...) it's obvious (again, with the benefit of hindsight) that they were all very inside-the-box. Apparently nobody even considered trying a different font.
This pg comment from that thread turned out to be particularly insightful, given today's news: https://news.ycombinator.com/item?id=211649