I've tried it three ways: having interviewees write code on paper, on a whiteboard, and on a computer.
Of the three, the computer fares worst. Many developers work in highly customized dev environments with special keyboards or monitor layouts. Using an unfamiliar setup, cold, in a stressful situation can make them sieze up or perform poorly.
Paper is no good because you can't watch their problem-solving process without hovering over their shoulder, and they get no feedback from you if they're going into the weeds.
The whiteboard seems to work best, because you get a good give-and-take effect between the candidate and interviewer.
For frogmetrics interviews, we did screen sharing sessions. You get to watch them code in their native environment, and see how they tackle the kinds of problems you'd be paying them to solve. In fact, the project we gave everybody was the next big feature we needed to implement, and the guy who won the job just brought his already-completed code along with him.
Does it sound right to you that your next big feature can be implemented during an interview, before the candidate is even hired?
I know, you didn't mean that it was all completed, but still... how much free work can you get from people before it's called abuse? If you had said "the guy who won the job just brought the best ideas with him", I'd be all happy.
1. We only used code from the guy who won, and he never saw the other submissions.
2. The interview project was time boxed to 24 hours. The guy who won did it in about 9 hours.
The actual feature is big, and has lots of edge cases: our devices need to be able to update themselves when new versions become available, roll back if there's an error, archive past versions, and never, ever end up in a state where they're bricked. The applicants did a small toy implementation of this functionality. The guy who won wrote good enough initial code to be used as a foundation for the rest.
They didn't say that the code from candidates that weren't accepted was used, just the code from the one that was. I suppose you prefer that interview coding tasks be arbitrary and wasted effort rather than put to good use?
Agreed. When having interviewees code, I really don't care about the result. I care about the process. I have always learned much more about the candidate from the discussion about the exercise than from the actual result.
I've found that trying to get candidates to write code on the spot is often more trouble than it's worth. It can take ages to cover any significant ground, and nerves are always going to affect people in different ways.
If I find a promising-looking candidate, I send them a few really simple tasks to code up (nothing at all complicated, or particularly "academic", maybe an hour's work at the very most). I set no time limit, and even don't demand all the tasks are completed.
This serves as a very effective filter for quickly discarding the completely-useless (which easily comprises 70% of the applications we get), and for the rest gives me a pretty good insight into their abilities.
I've tried sending these tasks both before and after interviews, and I'm definitely in favour of sending them first and interviewing second.
Apart from the obvious stupid-filtering benefits, I've found it helps immensely in keeping interviews relaxed. Having seen some code, I have a vague baseline for their abilities, so I can start with something I know they're familiar with and slowly work towards more complicated and interesting topics (without having to drop any questions that are met with a blank stare).
Although getting example code after an interview is an interesting experience. My predictions about a candidate's performance based on an interview have proven, with alarming regularity, to be way off the mark.
The setup I've found works best is to have two computers with browsers pointed to an Etherpad document, to allow pseudocode, and use problem domains that don't require the use of references.
That allows face-to-face communication, and avoids the uncomfortable violation of personal space induced by over-the-shoulder glancing.
Incidentally, this technique scales very effectively for phone interviews.
I turn into a blathering idiot whenever someone is watching me code. Because I have to spend like 60% of my brain power worrying about what the person is thinking of each intermediate step and if I should show it or keep it to myself.
But now that I started my own company, I get paid a little more than a guy of my age/qualifications should. Hopefully soon a lot more if I can keep building it up. I get paid based on the economic value of my products, not what any one individual thinks of me personally.
I know this problem. The last time someone asked me to write a sort in an interview I ended up implementing bubble sort. "You know", I said, "I remember knowing how better sorts worked, once. But I'm blanking on it now, so let me just write the stupidest thing that could possibly work, and then after that I'll see if I've had time to remember anything."
It didn't seem to be a big deal. Nor should it be -- the goal of your interviewer is to watch you do things like: test your code, remember to check the edge cases, manipulate pointers correctly, be able to figure out how your brain-dead bubblesort algorithm is likely to scale as N grows larger, knowing what exceptions are and understanding their proper role...
It's possible that the interview is better if you're missing your higher brain functions while you're writing the demo code. You're trying to demonstrate your everyday habits, the things you do instinctively, when your memory, your cram session, and Stack Overflow all fail you. It's like asking a trumpeter to pick up their instrument and blow a few notes: according to Malcolm Gladwell, some orchestra conductors claim to be able to detect expert players with nothing more than this one test.
ADDENDUM: Oh, look, Reg Braithwaite already said this, more or less:
It's more like asking a truck driver to read a sign at 20 feet, and to flex his hands and legs to make sure he's able to control the wheel and pedals.
The idea with writing code on paper in an interview is to disqualify the blind-since-birth truck drivers who would point to a rubber duckie in a line-up and say "That one! That is a truck."
It's actually more like asking a candidate airplane pilot to fly in a simulator.
I'm personally a huge fan of interview questions that require candidates to do tasks that are as close as possible to what they'll actually need to do on the job.
For devs this means coding, designing etc. For testers this means creating test cases and for a UI designer it means doing UI design. If someone made me interview an accountant I'd have them do some actual case questions for accounting. I wouldn't be able to tell if they did them particularly well but that's a whole other issue and is probably why no one has asked me to interview any accountants.
Like many other folks that interview using this strategy I've come to it after seeing how many people have a great resume but are completely unable to do the actual work. This is often true even for people that pass reference checks with flying colors. After getting burned a few times you realize the benefits of getting people to demonstrate their abilities before hiring them.
The counter point to getting developers to code during an interview (and presumably to accountants doing accounting problems in an interview) usually takes a form like "I can't write code in an interview setting b/c X, Y and Z but I'm a great coder under normal conditions".
I'm sure this is true for some people and it is unfortunate that those high quality people miss out on jobs they are qualified for. It's also a loss for you as a hiring manager. But as a hiring manager you're better off with an interview process that mistakenly screens out some good people if it ensures fewer poor hires that you ultimately have to fire. A poor hire is very expensive in terms of time, team morale, money etc.
So true. I learned this the hard way just like the author, by making some very bad hires.
I had always thought it was insulting to ask interviewees to write a simple two-line strcopy() function. Of course they can -- they've got a CS degree! So I would take that as given and move straight to more abstract non-coding questions.
It took me a long time to accept the fact that many people can talk about programming very glibly but can't actually do it. At first I felt like a jerk by making people write so much simple code in interviews, but it's led to much better hiring results.
part of me can't believe this was such an epiphany for this guy.
i've found the best way to screen candidates is to just ask them to sign up on project euler and work through at least three problems, in any language, and mail me the results. the quality of code i get so far is quite representative of the quality of the employee on the job.
yes, they could google the answers, but i disqualify the easiest-to-google questions (sum of multiples of 3 and 5 up to 1000, i'm lookin' at you). but honestly it doesn't matter - you can tell if they do because they can't explain it. and i can generally weed out anyone in the first place who would try to google the answer just from a skype conversation...
I have a similar completion rate and for a while I was really gung-ho about doing a Project Euler problem daily. Unfortunately, I discovered that there are sites which purport to have completed all 252(or maybe 253 now) and give out solutions.
That there are people who have cheated their way to high score, completely ruined the enjoyment of the site. It does seem a bit silly, that your own enjoyment of a problem is tarred by someone else's exploits. It shouldn't affect you, but it ultimately does, in some ways that someone else cheating in MMORPG affects one's own rat killing efforts.
That's really impressive. I'd interview anyone who showed such initiative. It's so common to interview "developers" who don't seem like they have any interest or passion about their work.
A 1000 employee company I know almost never makes hiring mistakes and has almost no turnover. One of the first things they do in the process is on-line assessment testing through a 3rd party. It could be for a specific skill or general aptitude. Score less than the 95th percentile and they disqualify you. They pay up front to eliminate the posers so that they can focus on legitimate candidates.
I've considered doing that, but I always worry that if I put too much of a barrier up front, I'll lose the superstars who know they can get a job somewhere else with less effort.
Is this company a very famous or desirable place to work? Up-front screening could work very well in that situation. But for less well-known companies like mine, candidates are less willing to go through a lengthy process to interview with us (except the desperate ones, and we don't want those).
I don't know if I'm a 'superstar', though I don't particularly favor the term, but I'll offer my own view on the onerous hiring process.
I won't do it. That is to say, I won't do it for just any company. I try to determine if a company has their act together by interviewing them back. I don't mind (and expect) some degree of white board coding or something, but there's a level of nuttiness that isn't worth my time for a company that maybe would be good, maybe not. If it's 'just a job' I have other options.
You can get away with more onerous hiring barriers if you pay a better salary or are otherwise a more attractive candidate (top reputation, etc) for employment. If you really are getting much better hires, you should be able to pay better I suppose. I think paying for code samples, or contracting for a few hours of work, etc, are also more respectful ways to get work from a non-employee and signals to the candidate that you're serious and they're at the final step.
Losing the superstars usually isn't that much of a problem. There are other superstars out there, and most of the top programmers I know care very deeply about being able to work with other smart people and not having to put up with incompetence. A rigorous hiring process is a plus in their book.
Besides, given a population of superstars that want the easiest interview process possible, vs. a population of superstars that want to work with other superstars, which one would you rather work with?
I'd be more concerned with the test not measuring what you want it to measure. It's possible to test well and still not be able to code yourself out of a paper bag, and it's possible (though somewhat less likely) to code well and not test well.
I recently started interviewing developers this way. A simple pre-screening interview weeded out around 80% of the candidates. I'm still amazed at how many people applying for a "senior developer" position can't do the most simple of coding tasks. Here's an "Ask HN" post I made about this:
FizzBuzz turned out to be an extremely simple problem (description in the post).
Atwood's article starts with a quote of yet another article with the following lines:
Like me, the author is having trouble with
the fact that 199 out of 200 applicants for
every programming job can't write code at all.
I repeat: they can't write any code whatsoever.
Surely, this is an exaggeration I'd thought... How can someone who calls themselves a programmer can fail to solve FizzBuzz even at under the pressure of being at a job interview?
To my horror, I discovered that a lot of comments to Atwood's post provide an incorrect solution and only prove the point of the article.
I realize that anecdotal evidence like this does not technically prove anything, however it does make me shudder inside...
That's why it is sometimes safer to hire developers with some open source background. You are able to actually read their code and (if you really want to know the person) their mailing lists messages.
I'd really like to know how to prevent incapable devs from even making it to the interview step. Obviously if you email test questions but it's too easy to cheat. What else can you do?
I email out Java code to use during the phone interview.
I require HR/recruiter/whoever to have the candidate receive my phone call from a place where he has email access. I call, introduce myself, and then immediately tell them that I'm sending out a couple samples of Java code. To avoid formatting hell, I PDF them (w/ Eclipse pretty formatting, even!). Then, we open up the code samples, and I ask questions about them. I can write more about this if anyone's interested.
Phone screen with not just your standard questions, but also simple algorithmic questions. An example might be to ask the candidate to write an algorithm to shuffle a deck of cards and describe it over the phone. This is sometimes a good indicator of whether they can communicate their solutions well without a whiteboard present.
It took the author of the article 10 years of interviewing candidates to realize this?
Ok, it's the author that sucks at his job. I'm sorry if that is not very constructive, but if I had to choose between having the horror show developer he describes on staff for 10 years taking up a cubicle and a paycheck, or the author on staff for 10 years hiring retards, I would take the horror show developer, as he would do far less damage in the long run.
Probably the author just has a positive view of people, and tends to see the best in people, or believes people when they tell him things. This would be fine for many jobs, but interviewing candidates requires a willingness to ruin someone's week. It is never the goal, but if a totally unqualified candidate shows up you must be willing to push them to answer questions so that you know that they are unqualified.
In any developer interview, you really only need to cover the following things:
- The person knows what they should know, based on their resume.
- If they say they were a linux administrator for 2 years, they should know all about linux basics and shell basics, they should be able to run a machine remotely without breaking it, they should not be dependent on any gui tools, they should know the common server software people run, and so on.
- If they were a web developer for 2 years, they should know about the stuff they supposedly doing, if it was backend code they should know about cache control headers and how ajax requests are or are not cached by the big browsers, they should know POST and GET and when to use them, and so on.
Anyone who doesn't know the things they 'should' know cannot be considered further, either they are lying on their resume or they are just turds who don't learn anything.
- They should be able to demonstrate an ability to code on the spot right in front of you. Some people claim they cannot code in front of someone else. Other than someone with an anxiety disorder, if a person can't code in front of another person it is just because they can't code at all. If their code is incorrect or suboptimal the main thing is that when you go over it they smack themselves on the forhead and seem annoyed that they would make such a mistake, and that they see right away how what they coded is wrong. If they can't get some challenging thing exactly right isn't such a problem as not 'being with it'.
- They should be sane and law abiding, not a drug user (and here I mean drug use which you can notice or that they *brag about), and that they have an ability to conform to some baseline social norms, such as bathing every third day or better, washing their hair and clothes occasionally, use and empty their trashcan, being able to talk to people without freaking out, and not going off on people or constantly insisting on doing things however they want to do it without strong rational each time.
This is basically 'smart, gets things done, can do things (vs coercing others to pick up their slack), makes a point to learn about what they are supposed to be doing, and isn't filthy or crazy.'
You may need to reread it again. Nowhere did I hear that it took him ten years to become a good reviewer.
In fact, the last sentence in the first paragraph is very telling.
In my early years I was very much hit and miss in terms of finding the right people and it wasn’t until a horror hire that I realised what I was doing wrong.
if a person can't code in front of another person it is just because they can't code at all
Never thought of it that way before, but it makes perfect sense. If you can ride a bike, change a flat tire, or balance a checkbook, you can do it alone or in front of others. Why not the same for programming?
I don't think it's intentional deception. There are a large number of 'programmers' who read slashdot and reddit and so on religiously. They probably hold a job where they are a developer of some sort, and they know about all the buzzy hot stuff out there from reading about it, but when they sit down to code they don't get anything done. Usually they manage to carve out a niche in whatever organization they are in doing non-coding tasks.
They are complete wastes of whatever resources they consume, but they are totally oblivious to this either for psychological reasons (ego defense or something) or the usual 'incompetent people have no way to detect incompetence', but probably a generous helping of each, ego defenses prevent them from integrating the memory of each failure they are responsible for (when they are fired from one job they aren't qualified for they just apply for another job they aren't qualified for) and the incompetence thing means they are unable to detect any real reason for their suckiness, leaving them with only the potential explanations they can come up with (usually stuff about how they were distracted or were having personal problems or had too many things going at once, as they are convinced that if they could just sit long enough and work at it they would eventually get it done. They are of course wrong, no amount of time would allow them to succeed at the task in question.)
if a person can't code in front of another person it is just because they can't code at all.
This is not true, as long as you're willing to wiggle on "can't". I do pretty poorly at coding while someone else is watching, because it takes most of my attention to code, and if there's a social situation going on, too, there's usually not enough left. The less well I know the watcher, and the better the watcher is at what I'm doing, the worse I do. This is true for finding bugs, too; having someone trying to find the bug on the same screen as I am means I'm almost guaranteed to fail to see it before they do. Fortunately, after 30 seconds or a minute of fruitless looking, most people will go back to their desk and let me actually get on with it, and then things get done. :)
I specifically made an exception for anxiety issues. In fact, it was the first several words of the sentence you are quoting. In fact, it is the only part of the sentence that you didn't quote.
Your ability to read the opposite of what is said intrigues me. Are you a birther by any chance? :P
I removed it to quote only the bit I was referring to, since I didn't think a phrase that seemed equivalent to "medical problems aside, ..." would matter; I wasn't referring to anything I'd characterize as an "anxiety disorder", or, indeed, any kind of medical issue at all. I'm also not quite sure how my explanation came across to you as a signal that I was unwell. :)
Either they're at a really small company where they are the only coder and where no one has the technical background to notice that they're just playing around with a VB gui editor and cutting and pasting in lines of code they found on the web. Or they're at a really huge company where they can hide behind an army of other developers or spend all their time doing peripheral activities.
