As Cletus wrote in his detailed comment, "this is the issue that will just never die." We discuss company hiring procedures here on Hacker News over and over and over, because most of us have applied for a job at least once in our life, and those of us who are building up a start-up have to think about whom we would hire.
The point in the submitted blog post by 37 Signals, "The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size," basically says that you should do a work-sample test to hire a programmer. And that's what research says. The more a hiring manager understands what a worker will do on the job, and the better the manager appreciates what a new hire may grow into doing after a few years on the job, the better the manager can devise a work-sample test to identify a good worker. There is a research base for this. Work-sample tests aren't perfect hiring procedures either--nothing is foolproof, and every hiring procedure goes wrong with both "false positives" and "false negatives"--but you can improve your odds of hiring a good worker by using a work-sample test routinely whenever you are hiring. As a job applicant, you can select better rather than worse bosses and co-workers by aiming most of your applications at companies that explicitly use work-sample tests as part of the hiring process.
With the help of several Hacker News participants, I have written a FAQ about company hiring procedures, revised over a few months of edits to fit the typical recurrent HN discussion of this issue. See
for the latest posting of that, with full references to the research literature and legal cases about hiring workers in today's world. Feel free to contact me through my HN profile
if you have suggestions for improving that FAQ before I post it to my personal website.
P.S. Even before I saw the prediction at the end of Cletus's comment, I planned to make the prediction untrue.
EDIT TO RESPOND TO FIRST REPLY ABOUT PUZZLE QUESTIONS: Yes, if a company in the United States insists on using puzzle questions as a hiring procedure, and justifies using those puzzle questions by saying that they want to see which applicants are "good problem-solvers," or "able to think on their feet," a rejected job applicant just might be able to subject the company to a very expensive lawsuit based on employment discrimination, unless the company has prepared beforehand a validation study showing that those puzzle questions have a bona-fide relationship to work requirements. I would not advise a company to take that risk, especially when the legally safer alternative of doing a straight-up work-sample test is available. The law is different in other countries, and as a reply in this thread points out, in the EU it is generally legal to use straight-up IQ-type tests in hiring processes, although those are underused by private companies in Europe, according to the sources in my FAQ post.
ANOTHER EDIT, TO LINK TO AN UNBELIEVABLE ANECDOTE ABOUT HIRING:
In August 2012, I heard a story from a hiring manager of programmers about the hiring procedure he uses as an initial screen for applicants who have degrees in computer science: "Write a loop that displays the numbers 1 to 100." That sounds awfully easy, even to me, but he says that the great majority of his applicants with accredited CS degrees fail that screening test. My earlier telling of the full anecdote
seem pretty nearly unbelievable in what they imply about how clueless many CS grads are, and yet I think the anecdote is a true description of reality. Cletus too mentions in this thread, with considerable agreement from reply comments, that many people hired as programmers cannot actually program.
All this teeth-gnashing regarding programming and logic puzzles are simply to cover up the simple fact that most people (even around here) are insecure about their programming skill, and more-so their intelligence. The fact is that these types of questions are indeed backed up by the very research you cite, namely that a work-sample test and general mental ability are the only reliable predictors of job performance. But this is exactly what programming puzzles are (and to a lesser extent logic puzzles). A programming puzzle is simply the intersection of a work-sample test and an IQ test. And of course, logic skills and programming skills go hand-in-hand. Instead of writing endless blog posts and internet comments about how useless these tests are (trying to convince a potential future hiring manager not to use them on you), just get better at them! You'll be better off for it.
Of course, not all programming jobs (most, in fact) need top 10% programmers with gifted IQs. It is important that jobs are honest with themselves regarding the level of talent they need for their next sexting iphone app. But this too is a part of the problem. Everyone believes their shitty app or website is going to change the world, and thus need world-class developers to help them realize their vision. Egos will be the death of us all.
> A programming puzzle is simply the intersection of a work-sample test and an IQ test.
No: a programming puzzle is the intersection of a work-sample test with one question from an IQ test. IQ tests get a good handle on your intelligence by testing you over a wide range of near-trivial questions, each known to similarly measure intelligence, and then adding the scores up. By giving you so many opportunities to succeed or fail, they don't fall victim to the "I just couldn't figure this one out" problem. Whereas programming puzzles do that all the time.
Now, if your interview consisted of 200 programming puzzles, each of which should take roughly 20 seconds to solve, then you could say it's getting a fair handle on someone's IQ as measured through the lens of programming. But throwing two 10-minute puzzles into the interview does nothing to help you measure anything. The "scores" you get are black-and-white threshold filters on a set of numbers (IQ) that are almost totally middle-grey.
> a programming puzzle is the intersection of a work-sample test with one question from an IQ test
In the worst case this is true. But ideally the question will have various components, and various levels of difficulty. I'm reminded of a blog posted here some time ago that had the candidate split a string based on a provided dictionary of words (or something to that effect). The best candidates produced maximally efficient code including memoization or dynamic programming. The worst failed to implement a basic "split on the space" functionality. Besides these two extremes there was a whole spectrum in between. The point is to not give questions that are Pass/Fail, but a question that has a spectrum of possible answers allowing one to find the limits of the candidates knowledge.
That still doesn't help the candidates who just don't see the trick.
What if you know memoization and dynamic programming, but it just doesn't occur to you that day, that to correctly split on spaces (not adding empty tokens at the beginning and end of the string, etc.) you need to use a state machine?
You could give the candidate that "step" of the solution and continue on to see if they know the rest, but the candidate will likely be feeling a loss of confidence. They'll think "hey, if I couldn't answer the 'easier' part, why should I be able to answer the 'hard' part?" and then their mind will lock up even harder when they search it for the next step.
This is why IQ tests are organized as a series of orthogonal questions. No question is dependent on any other question, so you can just move on quickly from a failure and prove your worth on another question.
In fact, on a usual IQ test, you're even given five or six "presentations" of each question, each slightly different--like math homework questions with different numbers inserted. It happens quite often that you'll get stuck on, say, recognizing how one specific pattern of shapes go together, but will be able to see a different pattern instantly.
Interviews do the opposite of this; they are presented as if intelligence is a linear quantity, and if you fail on any part of a puzzle, you couldn't possibly be able to solve a "harder" part. In other words, interviewers are using the "error" theory of intelligence, rather than the "bugs" theory[1].
>That still doesn't help the candidates who just don't see the trick.
This may be true, but at the same time candidates who DO see the trick get positive points. Because a lot of these puzzle questions simply require you to think outside the box, if you're forgive the use of that phrase, and honestly if you have hard programming problems to solve that is a killer skill.
I have (once) had the problem where my mind locked up on a puzzle question in an interview. (Disclaimer: Usually I'm really good at them, but then I usually get most of the IQ questions right too...) But I got the job anyway.
The point is that HOW you use these questions is at least as important as the content of the questions. If you ARE willing to work with the candidate on finding the answer, and you make it clear that it's not a make-or-break question, then I think it CAN be (at least) a positive filter for awesomeness.
Yes, sometimes you could let someone fall through the cracks that way. Nothing about the interview process is perfect, though, and if what you really need is someone who is awesome, then letting someone get away who IS awesome is much better than serially hiring people who AREN'T awesome hoping that you find one eventually.
That said, and as others have pointed out in comments: 99% of the projects I hear about on HN would NOT require awesome. They just require competence. And solving IQ puzzles isn't how you detect competence.
If your point is that these sorts of tests don't have any statistical power, I would whole-heartedly agree. Certainly you have to be careful with the types of conclusions one draws from the results of such tests. But they are not useless. The probability that a candidate will work out after having solved the problem with the best and most efficient solution is certainly much higher than if she only gave a middle-of-the-road solution. They are useful as positive filters.
And just to be a unnecessarily technical: if you accept the premise that the combination of appropriate CS knowledge and intelligence increases the likelihood that a candidate will successfully answer the question, then the fact that they answered the question successfully increases proportionally the probability that the candidate has sufficient CS knowledge and IQ (good ol' Bayes' theorem). This then increases the probability that the candidate will be successful on the job (citation in ancestor comment). That was a long winded way of justifying these types of questions as a positive filter.
I didn't say they're not a positive filter; just that they're a black and white positive filter.
Usually, hiring isn't about "passing or failing" candidates; it's about ranking candidates relative to one-another, and hiring the "best" ones (under whatever utility-function you think will help your business succeed.)
And putting people into "really impressed me in solving a puzzle" and "did not really impress me in solving a puzzle" does not much help in ranking people. It just gives you two buckets, one full of people who were both intelligent and lucky enough to get that particular trick that day; and one full of people who might or might not be intelligent--more intelligent than the people in the other bucket even--but who didn't get the trick that day. Personally, I'd rather not allow "luck" into my hiring process if at all possible.[1]
And this isn't so hard! I don't see why there is such a strong digging-in against the notion of changing away from "one or two long puzzles, that each give a single bit of informational entropy about the candidate's capabilities" to "hundreds of trivial, short puzzles, that together give a strong, statistically-sound measure of the candidate's capabilities relative to the other candidates measured."
I imagine the answer is that "the long-form questions give me a chance to get to know the candidate better, and see their [cognitive, and social] approach to problem-solving." But the OP (this guy: http://news.ycombinator.com/item?id=5264735) never mentioned knowing someone's approach to problem-solving as a dependent variable for their workplace success. IQ tests--relatively ranking someone's general problem-solving ability--is a dependent variable for workplace success. Work samples--showing, technically, that you can actually do the job as asked--is a dependent variable for workplace success. That's it. Rely on those, not your gut.
---
But, if I may go on a bit of a rant here...
This implies that convincing your interviewer that you'll be a nice guy who stays on-task, doesn't complain about problems, works long hours because they're "dedicated to the company", will talk down their successes (that is, be "humble") and won't demand fair compensation for their abilities, and is otherwise the "Agreeable, Conscientious"[2] kind of person that schools are expected to produce[3] and HR departments want to consume... is not a dependent variable for workplace success. There's no correlation. People can be great workers and produce their best work with their boss hating them and their coworkers thinking they're an arrogant jerk. And people will not just put up with them--but even approve of their behavior--if they get the job done better than someone who isn't like this. (Anyone who's ever watched House will probably grasp the concept intuitively.)
In fact, in software development at least, I suspect that this "not submissive to workplace domination" attitude can even be positively correlated with success--it's what gives people a sense of ownership and responsibility over project components, makes them passionate about making things right instead of just working, getting coworkers to refactor their own APIs so it doesn't hurt to consume them, etc. And I think that at least some software shops know this--it's, as far as I can tell, the original meaning (before it got diluted to meaninglessness) of looking for a "rockstar" developer. A clearer term might be developer primadonna.
But these types of people--unless they're lying--interview very badly, since, implicitly, an interview is an hour-long test of meaningless workplace submission. This probably isn't a bad thing--if you have the type of organization heavily infected with stress addiction[4], then this type of person--who will avoid the gametalk[5] that powers reciprocal dopamine transactions--won't fit in well anyway.
But if you're looking to build an organization full of people who care more about winning[6] than about "being a company", my advice is: skip the interviews. Just hand them the IQ+work sample test, send them into a room, and get them to do it. Maybe we could even turn this process into some sort of widely-recognized certification [a developer's journeyman certificate, basically] so the interviewed would only have to put up with it once. That would reduce the hiring process, simply, to "let's go out for a pint." And that sounds, in a healthy industry, like how things should be.
---
[1] For the same reason, I don't want to know what school you went to; luck [of parentage] significantly determines what schools you'll naturally end up at, with or without trying, but when people know what school you went to, a peculiar kind of nepotism/cronyism appears, with Harvard interviewers hiring Harvard graduates even if the Ohio State University candidate is otherwise equivalent, etc.
I appreciate your comment. I didn't realize you were actually advocating giving a bunch of small programming questions in place of one or two involved programming questions. I totally agree that this is a far superior way to test programming ability. Actually developing such questions would be rather hard I think, it seems it would be very easy to degenerate to trivia. Perhaps a set of questions similiar to that Quixey one minute programming challenge that was posted here a while ago (http://challenge.quixey.com/)?
I actually have a very ambitious idea to completely disrupt tech hiring. You have echoed much of my rationale. I don't want to go into detail, but the premise is basically to remove ego from the hiring process and make it far more statistically sound. Of course, this is also the very reason would very likely fail. Everyone loves to employ their pet "interviewing hack" to find world-class developers (who happen to be a mirror image of themselves). Most people will not give that up lightly.
The "loss of confidence" part, I don't see why that shouldn't be taken into account. If a candidate can't recover from failing, they're not likely going to do the same when they fail in their job. It also may indicate that the candidate is not teachable if they can't understand or have a hard time with your hint. Hints are a good way to progress an interview.
It isn't that people are insecure about their programming skill and intelligence. It's that they are irritated by tests which don't measure what they're supposed to. If any insecurity is involved, it's about ability to ace unrepresentative tests ranging from trivial nonsense to outright hazing.
You advise readers to "just get better at [the tests]". That's practical. But it goes right back to the problem, that many of the tests mostly index whether you prepared for that test.
Everyone talks about how many applicants are terrible at programming, or how many employees are mediocre. But at least offline, we rarely mention the complementary problem, that many companies are terrible at interviewing and hiring.
We talk as if being on the other side of that desk, in a position of power, means you are automatically good at interviewing. It isn't so. When a company treats candidates abusively - or takes bad hires who are extra slick, cheap, submissive, or similar to themselves - that isn't because of bad applicants. It's a failure of that company.
We can recognize that a defensive and condescending attitude is toxic in a programmer. But the same attitude in an interviewer is completely accepted. They're on top, applicants are on bottom.
> It's that they are irritated by tests which don't measure what they're supposed to
I disagree that programming puzzles don't measure what they're supposed to. Programming puzzles are a microcosm of what programming entails day-to-day. They test the same exact mental faculties. Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request. Doing these type of "low level" tests then allow you to include components that do measure one's mental flexibility and creativity, i.e. intelligence.
Sure, we can give a candidate a test that exactly measures what they'll be doing on the job. But what if the job changes? What if we need them to use a language or a framework outside of their comfort zone? This is the reason these tests are not specific to particular tools. You want a programmer that has the intelligence and command of computer science concepts to quickly jump between various tools with little effort. Comparing strings and reversing linked lists do absolutely measure this capability. No, its not testing whether you memorized how to do these things, its testing whether you have the mental flexibility to figure it out on the spot.
Comparing strings and reversing linked lists do absolutely measure this capability.
This depends entirely on the quality if the test. I suspect for many of these ad-hoc tests thrown together for an interview, the quality will be low and the overlap with other tests high, meaning they can easily be passed by someone who googled 'top 100 programming interview questions' and read through them.
Comparing strings or reversing linked lists are not actually things that most programmers would/should ever do themselves or even nowadays bother to learn or remember outside of a CS course, because the code has already been written in every standard library, but they are things that programmers might still be required to learn in order to pass interview tests, and you would thus be testing at many your candidates on whether they had memorised a few of the problems likely to come up, not on if they could work them out on the spot.
Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request.
I don't know what strcmp is in the way that you're using it. Does that make me a bad web developer?
Something orthogonal to this came up recently on Reddit, and it was observed that in the real world (anecdotally), 95% of professional programmers don't know what the Lyskov Substitution Principle is.
I think that most interviewers will happily give you the definition of strcmp if you don't already know it. If you're not a bad web developer, you'll happily come up with some code on a whiteboard that meets the definition.
Anecdotally, perhaps, 95% of professional programmers don't know the definition of the words, "Lyskov Substitution Principle". That's not the same as not knowing what it is.
Two weeks ago I interviewed at a subsidiary of a major web property, and "[w]hat is the Lyskov Substitution Principle?" was on a written test they gave me.
I would have told them that I had no idea what that was, but then told them what the "Liskov Substitution Principle" (named after Barbara Liskov of MIT) was. I mean seriously, if a company can't spell, do you want to work there? :-)
I think this is basically true. But a point worth making is that a "puzzle" problem is a much heavier thing than lightweight tests like FizzBuzz. If you hand someone a puzzle you're pretty much guaranteed (absent the rare, truly gifted candidate) to watch them struggle at a whiteboard for half an hour. FizzBuzz (or "reverse words in a sentence", or whatever) will tell you the same thing in a minute or two.
Basically, the Venn diagram regions for "can code fluently" and "can solve puzzles" have a ton of overlap. So test the first one, not the second.
You make a good point. The question is how much of an overlap is there? Is there value in adding a component of raw intelligence to the question (say, applying a data structure in a unique and non-obvious way)? It is reasonable to expect that the act of programming fluently entails a sufficient level of IQ such that there is nothing more to be learned from attempting to craft a question that tests intelligence directly?
Two bits of information could potentially answer this: does intelligence as a predictor of future job success drop off past a certain level? Also, what does the distribution of IQ look like for those who are fluent coders?
I would consider myself a very good programmer compared to the hundreds I've worked directly with - think good quality, clean, well tested code delivered on time that delivers outsized value in production.
I am however useless at these mental gymnastic and logic puzzle things.
Having a high degree of programming skill is weakly correlated with these kind of interview questions, if that.
> most people (even around here) are insecure about their programming skill, and more-so their intelligence.
> Instead of writing endless blog posts and internet comments about how useless these tests are ... just get better at them! You'll be better off for it.
This doesn't hang together. Will getting "better" at programming tests & logic puzzles make you a better programmer? Make you more intelligent?
If not, doesn't that illustrate precisely why people seem to have a problem with these tests? Namely, that they are primarily a measure of preparation & studying for the exam?
It depends on your strategy for "getting better at these types of problems". You can cram and memorize all the algorithms you can, and that may allow you to pass some filters that you otherwise would not have. This strategy has its uses: besides being offered more jobs, you now have knowledge of these algorithms in your head and thus more tools in your toolbox.
Ideally, the candidate would learn about these "exotic" data structures, understand their properties, learn the cases where they are useful, and most importantly develop mental heuristics to recognize patterns of problems and appropriate classes of solutions. This will also help you pass these filters, and in the process will absolutely make you a much better programmer.
Speaking for myself, I have purposely avoided studying specifically for interviews. My reasoning was that by relying on my existing knowledge + logic, I would give my interviewers better insight into my thought process and approach to problem solving.
If the benefits of spending time with this material are as significant as you say, perhaps I will revise my strategy (or, lack thereof).
Reading the summary you've linked to, isn't the implication for this thread that certain puzzle type questions, if they do a reasonable job of measuring general intelligence, are useful predictors of job performance? (With the big caveat that in the US they are potentially illegal.)
The popular response here of "I'm never going to solve these puzzles in my day to day" seems to wholly miss the point.
The claims about U.S. employment testing are completely wrong. This is how it works: there is an unconstitutional insurgency (called an antidiscrimination agency) that counts the number of negros on your staff, and if you come up short, they steal from you to prove how badass their homies are.
They generally leave high-tech alone. Firstly because high-tech will hire a pitch black Indian teenager with an outlandish accent if he can code. Secondly because there are not enough negro programmers from the projects to fill a bus.
In the unlikely event they do come after you for IQ testing, you what the large companies do: keep right on IQ testing, and appoint a director of racial sensitivity. His job is to use "sensitivity" and "nuance" to balance out the staff's "intangible factors". That's Newspeak for running the negro quota and making sure the quotees are selected for a pleasant personality and any sort of useful talent whatsoever.
The point in the submitted blog post by 37 Signals, "The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size," basically says that you should do a work-sample test to hire a programmer. And that's what research says. The more a hiring manager understands what a worker will do on the job, and the better the manager appreciates what a new hire may grow into doing after a few years on the job, the better the manager can devise a work-sample test to identify a good worker. There is a research base for this. Work-sample tests aren't perfect hiring procedures either--nothing is foolproof, and every hiring procedure goes wrong with both "false positives" and "false negatives"--but you can improve your odds of hiring a good worker by using a work-sample test routinely whenever you are hiring. As a job applicant, you can select better rather than worse bosses and co-workers by aiming most of your applications at companies that explicitly use work-sample tests as part of the hiring process.
With the help of several Hacker News participants, I have written a FAQ about company hiring procedures, revised over a few months of edits to fit the typical recurrent HN discussion of this issue. See
http://news.ycombinator.com/item?id=5227923
for the latest posting of that, with full references to the research literature and legal cases about hiring workers in today's world. Feel free to contact me through my HN profile
http://news.ycombinator.com/user?id=tokenadult
if you have suggestions for improving that FAQ before I post it to my personal website.
P.S. Even before I saw the prediction at the end of Cletus's comment, I planned to make the prediction untrue.
EDIT TO RESPOND TO FIRST REPLY ABOUT PUZZLE QUESTIONS: Yes, if a company in the United States insists on using puzzle questions as a hiring procedure, and justifies using those puzzle questions by saying that they want to see which applicants are "good problem-solvers," or "able to think on their feet," a rejected job applicant just might be able to subject the company to a very expensive lawsuit based on employment discrimination, unless the company has prepared beforehand a validation study showing that those puzzle questions have a bona-fide relationship to work requirements. I would not advise a company to take that risk, especially when the legally safer alternative of doing a straight-up work-sample test is available. The law is different in other countries, and as a reply in this thread points out, in the EU it is generally legal to use straight-up IQ-type tests in hiring processes, although those are underused by private companies in Europe, according to the sources in my FAQ post.
ANOTHER EDIT, TO LINK TO AN UNBELIEVABLE ANECDOTE ABOUT HIRING:
In August 2012, I heard a story from a hiring manager of programmers about the hiring procedure he uses as an initial screen for applicants who have degrees in computer science: "Write a loop that displays the numbers 1 to 100." That sounds awfully easy, even to me, but he says that the great majority of his applicants with accredited CS degrees fail that screening test. My earlier telling of the full anecdote
http://news.ycombinator.com/item?id=4603414
and his
http://news.ycombinator.com/item?id=4919749
seem pretty nearly unbelievable in what they imply about how clueless many CS grads are, and yet I think the anecdote is a true description of reality. Cletus too mentions in this thread, with considerable agreement from reply comments, that many people hired as programmers cannot actually program.