Hacker News new | past | comments | ask | show | jobs | submit login

If I were forced to reduce the entire problem of tech punditry down to a single sentence, I think I'd try this: "Internet time", the incredibly-accelerated pace of everything, is a modern myth, a lie we tell ourselves. In fact, meaningful change takes time, perhaps as much time as it always has.

Three years? Come back in, say, seven more years. By then maybe we'll know if Apple's store has a systemic, unsolveable problem or is just suffering growing pains from its overwhelming flood of success. Half a million apps and billions of downloads: It's a wonder that the system isn't considerably worse than it was a few years ago. Just think of how much stuff has needed to be scaled out and scaled up. They're probably running as fast as they can just to remain in place.

Apple's "problem" is that they have a fairly small team that can only accomplish so much at one time, and yet they allow that team to invent bold new things that are minimally functional, then release them, then succeed so overwhelmingly that it's a struggle to keep up with the traffic. But does anyone have a better idea? This is what innovation looks like.




    "Internet time", the incredibly-accelerated pace of 
    everything, is a modern myth, a lie we tell ourselves
I wouldn't call it a lie, because software scales better than humans do.

When you're saying "half a million apps and billions of downloads" it's as if Apple themselves worked on those apps and all of those downloads were manually packaged by Apple employees in envelopes, signed by Ive and sent by postal office. Well, actually, the number of downloads is irrelevant.

And if Apple can't handle the approval of those million apps, maybe that's because they've dug themselves into a corner by trying to force an asinine policy that doesn't even work for its intended purpose anyway.


As I read it, you're positing two things:

1) It's effortless to build an infrastructure that handles reviewing half a million apps and hosting billions of downloads, and to scale up to those levels from nothing.

2) The app store review process doesn't serve its intended purpose.

If that interpretation's correct, how do you support those assertions?

#1 is demonstrably false, given the effort that many companies exert to handle reviewing and approving numerous kinds of content, and then vending that content to consumers around the world. Netflix, Amazon, Microsoft, Apple, and Google are just a few companies that have large chunks of their organization devoted to approving content (be it movies, ads, apps, or ebooks) and then supporting the systems which host and vend that content around the world. Both are non-trivial problems and difficult things to scale at those magnitudes. To claim otherwise is disingenuous.

I'd also claim that #2 is false. Malware is not a problem on the App Stores, and I feel eminently confident as a consumer that I can trust apps purchased on Apple's App Stores. As a developer, the review process has caught bugs in my apps before they've hit my users, and they've also been quick to approve updates that address urgent bugs that slipped past both my and their testing. You may disagree with some of their policies and the review process does make mistakes, but I don't believe you can assert they're largely ineffectual or incompetent, nor do I believe you can claim the process doesn't offer benefits to consumers. What other software store is as confidently and easily used by consumers around the world?


it's as if… all of those downloads were manually packaged by Apple employees in envelopes

Sure, that part does scale. The fact that scalable processes can scale is not a myth. The myth comes in when we get so dazzled by those awesome new scalable processes that we handwave away the distinction between that which scales, and that which does not.

When you speed up one portion of a process you don't speed up the process, you just move the bottleneck. The slowest and most expensive part of shipping an app used to be printing and mailing disks. Now it's App Store reviews. [1] Or producing documentation. Or deciding whether the buttons work better on the left or the right. Or the volume of calls to your support line, or posts to your online FAQ. In whole sectors of publishing the bottleneck is marketing: Your book, movie, app, or platform can't grow faster than the number of people who can be convinced to buy it, and convincing people is a very human-scale process; to build trust requires attention, and attention is not a scalable resource. Despite modest advances in brain-enhancing drugs, we still think and feel at about the same rate as ever, and despite advances in font technology the number of words we can read per waking hour is about the same as it was two hundred years ago.

---

[1] Of course, the rate of shipping app updates depends on the speed of the review process, but the first derivative of the rate of shipping app updates depends on the rate at which new reviewers can be hired and trained, and that is even more of a human-scale process.


They are running out of places to stuff all their bags of money, but yet they "have a fairly small team"? Hiring is tough, I know, but it's a lot easier when you can pay about whatever you need to to get talent.


Hiring is one problem you just can't throw money at in order to solve.


My time in contracting work would argue against your point. 0-250 people in 30 days? No problem.


You're kidding! Would love to hear more about this point.. specifically including an address to the quandaries suggested in Mythical Man Mont.


I think there are four major points in this issue:

1) Bodies (or as we used to call them "cheeks in seats")

2) Qualifications

3) Quality

4) Non-linear scaling of work output.

When you deal with very large contracts, like say for the government, the customer is interested in two things: #1 and #2. #2 is viewed as a proxy, or a logical result for #3. And #4 is not well understood inside of most large organizations to this day as people are considered the same as equipment.

Six Sigma (6S) (http://en.wikipedia.org/wiki/Six_sigma) is more often than not the strategy used in these kinds of organizations.

To that end, if you consider a beer bottling factory, you can linearly scale the operation if you simply add more lines to the process. Substitute "equipment" for "bodies" in #1, #2 is the make and model of the equipment and #3 is irrelevant if #2 is correct and #4 is possible under these constraints and according to 6S, if done properly will linearly scale your output at no loss to quality.

What happens is that 6S is then applied to the world of people in big projects because we lack the management tools to measure and monitor large programs and 6S is as close as we get. Most important in this milieu is that what is measured (I'm being a bit reductionist in this statement but meh...) is purely the number of equipment (bodies) by the work output (lines of code/reports/successful missions/etc.) with very little assessment done on quality.

People are hired based almost entirely on resume bullet points. Usually with a cursory phone interview and if you're lucky a face-to-face interview so you can ask them to recite their resume bullet points and make sure they aren't crazy people. There is usually no other test. If you hire a person, so long as they don't violate some workplace rule, they fulfill #2. The only real determinant in who you hire is that the resume fills most of the checkboxes that the client has laid out during the program competition phase a year or more ago. Are they missing some points? There's usually a process where the resume is floated up to the client and they usually just sign some waivers that "yes yes, #1 is more important than #2 and we're willing to accept a minor negative in #3" so that the hiring agent doesn't get in trouble.

When a more senior manager wants to know how shop XYZ is performing with the 150 new people they just added to the contract, the report is invariably full of bar charts and numbers about such things, often with the important figures on different slides and almost no analysis that would demonstrate that the scaling is not working linearly. "We answered k requests for reports" on one slide "we are at 93% of hiring strength" on another and yet on another "we need an increase in budget for labor category q1.3 as we've had 50% turnover in that sort of position" etc. And that's more or less how the weekly or bi-weekly program management review PMR will go.

If all of the boxes are checked, the PMR was successful. If there were unchecked boxes, say "we are at 91% hiring strength instead of the goal of 93%" there will be hell to pay. Even if the 91% is more effective (produces more output) than the desired 93% because the quality is not under consideration because the bodies have met the qualifications for their position.

These are very manufacturing oriented metrics completely without regard to the nature of the work under consideration and they are used almost universally in large organizations when they need to quickly accomplish #1.

Now why does this happen? Why are potentially poor candidates hired? Anybody who's hired enough people knows that #2 is often a falsehood. People will put down any sort of crap on their resume. You have to vet people, often through some kind of testing process to hopefully ensure quality -- even if that doesn't always work, it's something. Some companies use education as a proxy for some of this: as in "you must have graduated from Harvard, Yale, Brown or Stanford for consideration". But the god honest truth is that in most of these organizations, the need to put cheeks in seats outweighs the desire to put quality people in the seats. Why? Subtle quality signals get lost when reported in aggregate to management, and thus management only cares about the kinds of things that aggregate information tells them. Your boss's boss's boss's boss wants 150 people on that contract yesterday so they can start realizing the revenue from the contract. By the time it gets to you, and your success/failure gets back up to him, the only thing that matters is that you push as close to 100% as possible. Not that you just hired a guy who can fit a chess playing algorithm inside of 12k of memory on an Amstrad CPC. Or somebody who speaks 12 languages fluently because the contact only requires that they know 2.

On the other side of the equation, when you win a contract and have to plus up on hiring 1200 people inside of six months, you will accomplish this or you will loose the contract. 1200 people is more important than 1200 fully qualified, high quality people.

And finally #4. As most people know #4 is still not well understood. I mean most people notionally understand that you don't just add twice as many people to a project to finish it twice as fast. But there are subtle unsolved problems with this. For example, can you produce twice as much in the same amount of time with twice as many people? How about, "well how many people do I need to add to double output/halve production time".

If these fundamental questions don't have solid answers, organizations will attempt to solve them by throwing metric tons of money at them because that will give the appearance of solving it. Subtly in the higher ranks of the organization, they will ensure that the information reported to them will ensure that it looks like it solves it.

I'm not saying this is right, but money does in fact solve hiring. From the organization's standpoint they need to triple their output? They hire 3x the number of people. How much does this cost in salary per person? Supply that much money.

Output didn't triple? Add more people!

Done.


Your specific claim would only make sense to me if I could think of an actual reason (other than a systematic mindset problem) they it could take 3 years for Apple to change improve these things.

Sure, the Internet community is a bit impatient, but it's also true that it's quite possible to make drastic improvement in a lot less than 3 years.


It's very slow and difficult to change something that is working. It is even more difficult to change something that is both working and growing like gangbusters. And the larger something grows, the harder it is to change.

(Ironically, things with no customers are easy to change. And things that are obviously broken are also easy to change: It's broken, the worst that can happen is that it becomes twice as broken, so just go for it.)

There's a famous aphorism attributed to Napoleon: "Never interrupt your enemy while he is making a mistake." Rule zero of systems engineering is the inverse of this: Never interrupt yourself while you're succeeding. Do not kill the golden goose.

So, the first thing that has to happen for Apple to make significant changes in their process is that they have to stop succeeding so darn much. While the App Store is working they're unlikely to change the process. And, indeed, it doesn't seem like much has changed: They've fixed obviously broken things like months-long review wait times (as I said above, fixing obviously-broken things is relatively easy) but they haven't really changed the fundamental game plan. Why should they? It's raining money.

There are several reasons why it takes so long to improve working things:

Entire companies are built on the existing Apple model, broken as it may be. Break their revenue stream and these companies will scream and yell.

It's easier to swim with the tide of institutional inertia than against it. If you have to hire one hundred new App Store reviewers per month, those people need to be trained. There is (at least at first) no textbook for that job, or perhaps the manual is two years out of date. And nobody reads manuals anyway. And some important parts of any job are not written down anywhere. So the new folks will learn by watching over the shoulders of existing App Store reviewers. And so it goes: As things scale up, it becomes easier and easier to let the existing culture maintain itself, bolting on new parts as necessary, rather than try to change that culture.

When you want to change a working thing, it's not enough to show that your new version works. You must also demonstrate that it's a significant improvement over what you're doing now. This can take a lot of time. Science is patient work. If you're lucky you can run continuous A/B tests of small changes, and thereby gradually inch your way forward to a brighter future, but how do you A/B test App Store features when the developer community has a clue and a forum? Offer a feature to one developer on Monday, and by Tuesday night all the others will be at your metaphorical gate with metaphorical torches and pitchforks, demanding the feature for themselves. You have to go very slowly. It pays to do so below the radar.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: