Before the interview, I ask them to write some code to access an HTTP endpoint that contains exchange rate data (USD, EUR, GBP, JPY etc.) in XML and to parse and load said data into a relational database. Then to build a very simple HTML form based front-end that lets you input a currency and convert it into another currency.
I ask them to send me either a link to a repository (Git, SVN etc.) or a zipball/tarball. If the job specifies a particular language, then I obviously expect it to be in that language. If not, so long as it isn't in something crazy like Brainfuck, they have free range.
If the code works and is basically sane, that goes a long way to get them shortlisted.
During the interview, I'll pull the code they sent up on a projector and ask them to self-review it. If they can figure out things that need improving in their code, that weighs heavily in their favour. Usually this is things like comments/documentation, tests, improving the structure or reusability. If it's really good, I'll throw a hypothetical idea for refactoring at them and see how they think.
The reason this works is that, despite Hacker News/Paul Graham dogma to the contrary, "smartness" isn't the only thing that matters in programmers. It's actually fairly low down the list. When hiring programmers, I want people who are actually able to do the daily practical job of writing code, modest and self-critical enough to spot their own mistakes, and socially capable to actually communicate their decisions and mistakes to the people they work with.
I interviewed a guy who was intellectually very smart and understood a lot about CS theory. I asked him why the PHP code he sent me didn't have any comments. "I don't believe in comments because they slow the PHP interpreter down." Sorry, he can be smarter than Einstein but I ain't letting him near production code.
I love your interview style: actually test for things the dev will actually do.
I've applied to other jobs who's owners frequent HN, and they did things like test my ability to write sort algos or data structures... but the job called for a good knowledge of JavaScript, Backbone and general knowledge of django to build a mix between a CMS and an elaborate voting system.
They didn't hire me. I'm utterly trash at working on algos under pressure. I felt like a lost child just trying to write a simple sort during the interview. Worse, I made it harder on myself by asking to not write the code in MSWord (or some other similar thing) and instead use jsfiddle. The owner looked at the code and said 'Yeah, that looks about right to me, it should work.' I foolishly hit run.
What I learnt was two-fold:
1) this doesn't test my ability to do the thing you want to pay me to do (as proof, I'm writing more complex applications today at a job I enjoy far more than I would have at that company).
2) The employer can't differentiate between a "smart developer" or "average developer" until the script executes.
I went for a job a few months ago where I had to write a program on paper to produce report data from raw data collections, it was reasonably complicated. This is the type of thing I breeze through when I'm sat alone at my desk, but for whatever reason I just went into panic mode and couldn't think straight at all. I apologised and told them just that. It was frustrating to say the least.
The interview went ahead and we ended up talking about various architectures, design patterns, programming languages, functional programming and even HN, but I'm sure I never got the gig because of the technical test.
For the interview at my current gig they walked out of the room for 10 minutes while I had to write sorting algorithms and I was able to do it without a problem. So, for me at least, the element of being watched while I wrote the code seemed to be part of the problem.
> "I don't believe in
comments because they slow the PHP
interpreter down."
For some bizzare reason, a lot of PHP devs obsess over mini benchmarks and tiny speed issues while ignoring massive latency factors (like the DB).
I find it amusing, though, as PHP hasn't been a direct interpreter for years. It compiles to an internal opcode array and interprets that. While its opcode compiler is silly (it doesn't build an AST and directly emits opcodes, which causes all sorts of things not to work), it does have a proper lexer and I think comments are completely ignored aside from doc comments[1] which can be obtained in userland with reflection.
> For some bizzare reason, a lot of P̶H̶P̶ devs obsess over mini benchmarks and tiny speed issues while ignoring massive latency factors (like the DB).
This sort of behaviour isn't unique to PHP Devs, not even close.
Reminds me of all the roadies (road cyclists) who would come into our bike shop and demand the lightest components to put on their bikes, then rave about how they cut 10 GRAMS off of their bike weight. Totally ignoring the fact they're pushing almost 220lbs themselves.
Hah! I have this discussion every time somebody tries to upsell me at the bike shop. I tell them that if I'm going to try to take 5 pounds off the me+bike combo, I'm not starting with the bike.
Way back when, I did a small study (4 cyclists, 5 reps). The question: Could the rider tell which was heavier between a water bottle filled with lead shot, and one with glass beads. (2.5kg vs ~ 1kg).
1 rider managed to get more than half right. And he was a frame builder riding his own bike.
And there's nothing wrong with having optimized code. This seems to be a discussion about having the right priorities, not on the virtues of the optimizations themselves.
No, it's not unique to PHP devs. I've encountered C++ programmers that obsess over pass-by-value versus pass-by-reference and strings and minor things like that, even after that code has been written and proved to be fast enough, while ignoring the real bottlenecks that demonstrably hurt performance. A lot of times when there's a big glaring bottleneck, it needs to be solved at the macro level, by reworking the approach to macro-level features, seeing how you can get to the same place by doing fewer time-consuming operations. Those folks are penny-wise, pound-foolish.
> I asked him why the PHP code he sent me didn't have any comments. "I don't believe in comments because they slow the PHP interpreter down." Sorry, he can be smarter than Einstein but I ain't letting him near production code.
This one wasn't smarter than Einstein. It's been a while since I've been near a live PHP installation, but IIRC running with an opcode cache like APC probably means the comments will be parsed at most once (every server restart?)
This true - but without the opcode cache the slowdown would be minimal since all the comments are ignored.
Plus, the comments are there for future developers to use as a reference. Nothing worse than having something break, cracking open a file and have to spend hours having to untie someone else's code.
I'd rather have copious amounts of comments, then a minimal speed gain due to not having them at all.
Amen, brother. As Martin Fowler writes, "Any fool can write code that a computer can understand. Good programmers write code that humans can understand."
If Mr. not-Einstein had actually measured some production issue that could be traced back to commenting, he could always have written a filter that stripped them before production. My guess, though, is even he didn't believe this excuse; it was just a glib thing to say in hopes of wiggling out of trouble for being a negligent coder.
That's the thing: if he'd said "I was cranking this code out quickly, but, yes, it could do with some comments", I'd have been fine with that. The excuse is worse than what he's trying to excuse!
Since you bring up Fowler, in his book Refactoring he gives specific examples to show why you should AVOID comments. In most situations you should try to write code that is so clear no comments are needed. That's what the above code is really about.
I suppose someone could incorporate a procedure which strips down unnecessary strings from code files ("minifies"), especially comment blocks, during the deployment process, thus maintaining a nice development codebase while not having questions over performance.
Depends on your definition of "smartness". In my book the ability to assess the importance of things and their relevance to given task and ability to prioritize accordingly is very important aspect of being smart. The guy who 'does not like comments because they slow the PHP interpreter down' does not seem to be that smart.
And the mundane task of coding is probably not the most important part of any developers job. Understanding fully what the spec/task requires including covering the use cases the designer forgot to mention.
Sounds like a good interview; you're giving them a chance to have a go at something realistic using their typical toolset instead of a whiteboard.
I don't understand your point about dogma. I think HN assumes rationality as implied by smartness. Nobody wants to hire a poor coder/communicator to write code, CS degree or no. I bet most readers would find your process to be pretty good and have similar processes of their own.
I've also interviewed lots of intellectual "CS theory" types. PhDs, etc. In most cases, they have not worked in real software engineering environments that necessitate comments for simple HTML forms. Just IMHO, I don't think it's a huge knock, as some of these candidates can turn into game changers for your company.
I really like this. I'm finishing graduate school and I can't tell you how disinterested I am in a company when I apply or send an email and they want to ask me math problems or puzzles....And sadly, more ask puzzles than don't
You wanted a ______ and you are interviewing for a puzzle-solving mathematician really doesn't sound right.
I think it's only good if they fess up to it. Leveraging other people's work is fantastic, and underdeveloped in our educational system. Taking credit for other people's work, well.... :-)
"smartness" isn't the only thing that matters in programmers.
Smartness is overrated but experience is critical for speeding up work. I can implement a whole site in Django but without good experience I will be slow (even being good at Python)
Nowadays, I think we need to find new ways to evaluate candidates productivity. For example, just following a Q&A method (stackoverflow) I can be faster than other developers.
I'm with you on this. I always do pair programming interviews when I'm hiring. People may be able to bullshit me when I'm asking questions about their resume, but they cannot fool me, the compiler, and the unit tests when we're coding together.
Smartness is great, I like smart. But I agree that it's probably the fifth- or sixth-highest-weight factor in my list.
Personally, I think that if you want a brilliant mathematician/problem solver, then that's what you should put in the job listing. Then asking the kind of questions exemplified in the article is likely merited. On the other hand, if you want a good programmer you should mainly look at the quality of the code they produce.
We interview people this way where I currently work. I'm always surprised at the people who send stuff in that flat out doesn't work. There is no time limit, and the task isn't very hard. I actually consider it a type of fizzbuzz.
This works really well for the candidate too. They walk into an interview with a known quantity and can "own" the conversation by discussing the reasons that they made the decisions that they did. It ends up being a very context-specific portfolio.
Are you a prof at UIC (or alumni)?
I saw a job posting that basically described the first part of this, which made me smile because it actually made sense, instead of abstract nonsense about goats crossing bridges...
No, I'm a freelancer in London. The hiring process was for an organisation I volunteer for who wanted to hire a coder. I didn't want to work there, but I volunteered to help out with interviews and CV screening.
despite Hacker News/Paul Graham dogma to the contrary, "smartness" isn't the only thing that matters in programmers. It's actually fairly low down the list. [...] I interviewed a guy who was intellectually very smart and understood a lot about CS theory. I asked him why the PHP code he sent me didn't have any comments. "I don't believe in comments because they slow the PHP interpreter down." Sorry, he can be smarter than Einstein but I ain't letting him near production code.
That isn't what Paul means by "smart." Worrying about trivial inefficiencies like that is a sign of an ineffective coder. That's the opposite of a great hacker.
So, smart includes some kind of experience-derived common sense that isn't measured well by IQ tests...? Perhaps someone should tell the people who want to use IQ tests to measure the quality of programmers...
Paul Graham uses the word "smart," essentially, to mean someone who "works smart" rather than simply "working hard." In other words, it's someone who won't build a 12-foot fence if a 6-foot fence is sufficient. And it's someone who's capable of realizing as much in the first place.
This isn't "common sense," per se. It's more like uncommon sense. Most people follow an effort in -> result out paradigm, believing the two are perfectly correlated. A "smart" person, in PG's view, attempts to min effort and max result. (I've also heard this sort of hacker described as "lazy," in a positive and non-derogatory sense. A "lazy" person finds the most efficient and effective ways to do things, minimizing man-hours and resources.)
The relationship between "smart"/"lazy" and IQ hasn't been studied all that well, but it seems plausible that there is some sort of relationship. Perhaps it's a strong one, and perhaps it's not. Either way, I don't believe an IQ test as a single-factor qualification gate will filter effectively for "smart"/"lazy."
I use the word efficient instead of lazy. If I can do x and 1 other thing if I do x efficiently, why not strive to find as many things I can do efficiently as possible?
In this case, the word "lazy" is being used as a sort of tongue-in-cheek reference for "efficient." I don't think it's meant to be interpreted as being literally lazy.
1) I think a lot of start-ups want to hire "smart" people. Because they expect the new person to eventually wear many hats. Objective-C, Java, Android, CSS, server side concurrency, monitoring. An we've all seen Hunter and Schmidt reference that tokenadult usually posts when talk about interviewing comes around and it does seem that a general mental ability test (like an IQ test) combined with a work samples seem to predict future performance of that employee. Well except that one can't just straight up give IQ test to job applicants (there is a court case about that). So we are left with a job sample (which many forget to give, as is the point of the author). But instead many focus on the GMA and create proxies for it -- cute little puzzles about blenders, round manhole covers, and other such silly things.
2) Those interviewing don't know the technical stuff and are afraid you'd out-bullshit them. "How does an Ajax request work" well if the interviewer themselves doesn't quite know the details the might not be able to evaluate it properly. They could have it written down but well, some technical questions have many different levels of depth that a candidate might descent to. So a quick written answer to the question might seem wrong but it is really because the candidate is more advanced. So puzzles seems to be a generic and "easier" to handle.
I think (1) is probably the primary reason, along with the fun factor I mentioned above.
But honestly, why can't we just start giving IQ tests? Or at least asking for SAT scores? We shouldn't have to run through training puzzles just to prove we're smart enough to build a site.
> But honestly, why can't we just start giving IQ tests?
It's illigal. Or at least very very legally questionable.[1] Theoretically good proxies could be attacked in the same way, but there's a difference between doing something that could conceivably be the basis of a lawsuit and doing something that the Supreme Court has specifically ruled on.
I meant that mostly rhetorically in that I think that was a poor decision, or at least applied poorly. (Correct me if I'm wrong, but if you can show a high IQ is necessary for the job, isn't an IQ test acceptable?)
Still, asking for SAT scores is still legal and they correlate decently with intelligence.
I suspect the actual correlation is not with wealth (i.e. giving money so somebody with low SAT scores would not actually improve their scores - unless he would have enough money to bribe whoever is overseeing the test of course ;) but with some things that correlate with wealth.
It sounds like you heard "causation" when parent said correlation. Correlation is observed in past samples; what you describe is a mechanism for trying to disprove causation, but it won't remove any correlation from the past samples, and it probably won't remove the correlation with historical wealth of the family etc.
Parent said "primary correlate". Primary implies some special-ness, that is not like the others.
I do not see any way "historical wealth" could directly - without intervening variables - influence one's SAT scores. Of course, having affluent parents may mean certain value given to quality education, certain amount of care and access to development tools and so on - but then those should be primary factors considered, not wealth per se.
Money buys access to tutors and training materials. It buys access to free time for practicing for your SAT. It also buys opportunity to repeatedly retake the SAT until you get a grade you like.
"Does test preparation help improve student performance on the SAT and ACT?
For students that have taken the test before and would like to boost their scores, coaching
seems to help, but by a rather small amount. After controlling for group differences, the
average coaching boost on the math section of the SAT is 14 to 15 points. The boost is
smaller on the verbal section of the test, just 6 to 8 points. The combined effect of
coaching on the SAT for the NELS sample is about 20 points."
So in fact, it is more likely that people from wealthy backgrounds are able to better utilize their intelligence, and thus retain their wealth and status, whereas people from poor backgrounds are at a disadvantage when it comes to escaping their position in life.
Did you read more than half of one sentence that I wrote? The poor are at a disadvantage when it comes to escaping their position in life; if you need this spelled out for you, the poor are disadvantaged when it comes to making decisions about saving money that could help them improve their financial situation. Wealthy people are better able to make rational decisions about their money, which helps in retaining wealth.
Maybe you should actually read the article I linked to, if it would help you understand what I wrote.
If you are hiring for start up, you do not care why they are dumb. Potentially you can save money by hiring non traditional candidates and paying them less, but any advantage you get this way will be short lived. Unfortunately diamonds in a rough are rare and finding them cost more than their value.
I was originally replying to a comment that claimed that people became wealthy because of their intelligence. My point does not have much to do with the issue of startup hiring (admittedly making the thread a bit off-topic for the article).
I understand the position very well, and I think most people with common sense do. Of course it's harder to become wealthy than it is to achieve other positions in life.
I just think it's a stupid use of "disadvantaged." It implies someone else has an advantage, in to quote you, "escaping their position in life." It's like saying a bad baseball player is at a disadvantage when trying to become a good baseball player. It almost doesn't make any sense.
Same goes with the idea that poor people are disadvantaged about making decisions wrt money. Of course they are, or they quite likely wouldn't be poor any longer. To use baseball again, or any skill really, the experts of course have advantages in deciding the best course of action, and typically have more ways to achieve it.
What I've gleaned here is that you consider poor people to be novices at the skill of acquiring wealth. I agree. For some reason people seem to think it's far different from rookies in other skills, because acquiring wealth is perceived by most to be a vital part of success, happiness, health, etc.
Maybe they aren't as dumb - at least in the area that made them rich - as it seems. I can imagine someone bing smart in the university hall and dumb on the street, or vice versa.
oh really if the US had a system like we have in the UK the house of commons would have eaten quite a few high profile US polticians (including some presdents) alive.
Id love to see Paxman interview sarah palin or Ted Cruz for example.
Aren't intelligence and wealth correlated with each other, and both are correlated with SAT and IQ tests? I thought the explanation for the mysterious "g-factor" in intelligence that IQ is supposed to measure isn't that it exists in and of itself, but that a number of measurable outcomes (income, wealth, test scores, GPA, job prestige, etc.) are all positively correlated, and then that correlation is called g.
We need to define some terms. Education != intelligence. The IQ tests are silly and heavily judged as nonsense. These tests are focused on math mostly, but math alone is hardly the only area that matters. However, in this context, math matters, because in many ways it is correlated to programming.
I'd guess that there exists a causal relationship between "intelligence" as measured by standardized tests and the sum of dollars poured into someones education.
Standardized tests like SATs and school grades are a terrible way of ranking a potential employee and have significant biases such as income and age.
I didn't have very good grades and my SATs were ok but not great, both primarily because I didn't study or apply myself. I was completely uninterested in school. Get me to write code and you'll see what I'm capable of. I'm certainly not unique in this regard.
What kind of IQ tests? Math thinking? Language? Other? These tests are kinda old and don't prove anything besides the fact that the person has a certain level of intelligence but that doesn't mean they can code.
> What kind of IQ tests? Math thinking? Language? Other?
Both. They're not perfect, but I'm going to be picky and look for people who can generally reason well (both mathematically and verbally).
> but that doesn't mean they can code.
That's why you pair it with work samples.
The ideal hire would be an impressive GitHub + evidence of intellect. The GitHub shows they have coding chops, while an IQ test usually means they can react quickly and learn new skills.
That is an extremely long post that says "hire the rock stars". Except, of course, we all cannot only hire the very best, transformative people (like Jeff Dean at Google). And, he has no way to actually identify them, except "ask around".
It's entirely unrealistic as a business strategy. It sure is nice if you luck into it.
I agree with the principle of wanting to see code but keep in mind that expecting Github (or similar) is a bit ageist. I definitely went through a phase of my life where I coded like crazy and produced lots of interesting side projects, but that was long before Github or even Sourceforge existed. So I could point you towards some interesting stuff I passed around the BBS world in 1985, except that even I didn't bother to keep a copy after its usefulness faded.
I still have a bit to show on Github and mean to put up more when I have a chance but still, most developers find that when you've got a decade or more of experience, you start to want to take on new challenges and experiences (whether family or woodworking) in your free time rather than more programming.
What if all your most relevant work is owned by previous employers? Restricting hires to people with decent personal projects and/ or extensive open source contributions isn't ideal either.
I'm fine with only hiring people who have done some work outside their workplace. If they're really great at programming, they have the drive to try new things outside work. Indeed, a couple good personal projects is probably the best predictor of skill you could find.
(Skill and interest tend to be highly correlated.)
I used to believe that as well. Now I'm not so sure, mainly through my own personal experience.
For 5.5 years (2004-2009), I was a core developer and maintainer over at Xfce. Over that span of time I wrote tens of thousands of lines of code, perhaps more. I did this in addition to my job, which morphed from engineering project management to programming tasks that I didn't find very challenging.
After I left that job, I joined a tiny, early startup. The programming work there I found very challenging, and the insane hours quickly ate up any spare time I might put toward Xfce. Eventually I realized that lacking a life outside of work made me sad, so I left. Fast forward a little bit, and now I'm at another startup, which (while demanding, challenging, and rewarding) gives me time for outside pursuits and a social life. But I don't feel the desire to work on personal coding projects anymore. The job fulfills my desire to write code, and outside of that I'd rather be, well... outside.
It's interesting, because all my Xfce contributions are on git.xfce.org, and not on GitHub. My GitHub account contains 2 personal projects: a semi-finished Android UI library (which was really only relevant and useful in the 2.x days), and a script for building a Gentoo image for a Raspberry Pi (which I've since abandoned as I mainly run Debian nowadays). Looking at my GH account would be fairly unimpressive, and might even be worse than not having a GH account at all.
So I'd easily fail the GitHub test. If a particular company was making that a no-go for getting through an interview pipeline, well... their loss, I guess.
I'm responsible for a fair bit of hiring at my current company, and I rarely ask about personal projects anymore, unless a candidate has one listed on their resume. While I'm not always happy with the coding problems I usually give (they tend to be data-structure/algo problems that I'd probably have trouble solving in an interview setting), they do at least clearly show whether or not the candidate can write code, and I also get a rough idea of how cleanly they code, not to mention insights in how they approach and solve problems. And not being able to completely solve the problem isn't an auto-fail in my book either, as long as they tried to work on it and their approach (and whatever code they did write) was sound.
I do very strongly believe in the aptitude test + work sample formula as the best way to evaluate candidates, but the problem is more that it's not always so easy to apply those tests in a traditional interview setting. Maybe that just means we need to come up with a different way of interviewing, though.
I have a bunch of personal projects, but few of them are on github, and the ones on github are not done under my real name, because I find the concept of me having to work on nights and weekends to impress future employers offensive.
My personal projects also include things like my very playing around with a new language (so it is freaking ugly and so I would never want it used as a job indicator) and old abandoned projects that I've salvaged and updated as a favor to friends, which means the entire architecture is alien and "I Don't Care If It's Ugly Just Make It Work Because Right Now It Does Not Work At All" is exactly what the 'client' wants.
I also reverse-engineer game protocols and find vulnerabilities in them. There is no social good to me releasing that stuff. (No, I really don't feel like being a soldier in your Full Disclosure army.)
I do lots of coding stuff in my spare time. I don't do it to get a job, I do it because it's fun.
>> For 5.5 years (2004-2009), I was a core developer and maintainer over at Xfce.
>> But I don't feel the desire to work on personal coding projects anymore. The job fulfills my desire to write code, and outside of that I'd rather be, well... outside.
Another thing to remember is that you're 5-10 years older then you were at that point. Life priorities have changed. You might not be a caffeine-fueled 19 year old anymore, instead you're 28 with 2 kids and a mortgage. Or 34 and just have other things that interest you than tweaking Linux or exploring video drivers.
Sure. The intent of that statement was kinda muddled. I was more referring to the fact that my "main body" of open source code, to which my contributions stopped 4 years ago is all on a non-GH site, so it's interesting to look at what is on GH, which represents what I've done out in the open since then, which is... not much.
My assumption (which maybe is wrong) is that 5 years of OSS contributions that ended 4 years ago may not be all that impressive. Why did I abandon it? etc.
The github test isn't about your what you have on github. It's about what code you can show, out in the public. Github just happens to be the common way to do that now.
If a company requires Github specifically, than it's just a stupid company you wouldn't want to work for anyways, so it saves you the trouble.
You'll probably never get anyone who used to work at Google/Facebook/Apple/Twitter then.
The problem is that IP agreements in the state of California say that the employer is not entitled to work done "on the employee's own time, without using the resources of their employer, and unrelated to the lines of business of their employer". Google claims to be in basically every tech-related business there is. Do something as simple as a mobile app or casual game and you can potentially get in trouble for it because it may compete with a division you've never heard of.
There's a procedure to get around this, but it's fairly slow and bureaucratic, and so many Googlers find it more efficient to do their "for fun" coding as 20% projects. Facebook has a similar system with their hackathons. Apple just works its devs so hard they don't have time for independent coding products.
I think that GitHub profiles are a good way to identify good devs who don't work for the leading tech companies, but once you're in one of them you a.) don't have time for personal programming projects anymore and b.) find that the stuff that your coworkers are doing is more interesting than pretty much anything you could work on as a solo dev.
I like spending time with my friends and family, some of which I have to travel far to see, for one.
I spend 40+ hours a week in front of a computer, when I get a chance to go out in the world, I want to GO OUT.
None of these things don't mean I don't love software (I do) or I'm not interested in it (I am). I just have varied interests.
Disregarding that, I used to work for a Big Nameless Faceless Corporation, and the legalize in employment agreement said basically that because I was a software engineer for them, they owned all the software I wrote while employed, even if I wrote it on my own time and with my own computer.
I've heard one too many times that I have to react now and voice my confusion. When you have students and you want then to have had some work done outside of coursework, I'm perfectly ok with that as I understand the merits but I wonder is it true for people 10+ years into their career. I see great people having great jobs who, if they have free time, code something that can help their current job but that code also ends up being own by the employer and is very rarely released as open source.
GitHub is a false profit. Almost every excellent programmer I know has almost nothing in their public GitHub. I've been doing this for a great many years however going off my GitHub I'm barely a programmer if that's the metric you're tracking.
Management consulting firms and Wall Street banks have employed somewhat strict GPA and SAT/GMAT cutoffs for quite some time, but even they can't resist the allure of brainteasers during the interview process.
Because SAT scores are a very very terrible indicator of if you have 1) the ability and 2) the work ethic to build a site.
And how old are you? SAT scores? I'm only in my 30s and I don't remember my what my SAT scores were or remember if they were "good" or "bad" I only know that they changed the SATs after I took them because my youngest sibling took a different one than I did, with some crazy scoring. I wouldn't be able to find my SAT scores even if I wanted to.
In what way will an IQ or SAT test give you an indication if the applicant even knows what a compiler is? Your comment is cute but I'm not sure how useful it is ... :(
1) I think a lot of start-ups want to hire "smart" people.
What does it really mean to be smart? Lately I cannot stop thinking about it. I have always been considered a 'smart person'. I am a self-taught freelance developer now - it used to be my hobby and somehow (mostly because I needed location-independent work quickly) it became my profession. I get by because everybody thinks I am smart but I feel like an impostor because the more I think about myself the more I realize that intelligence is not some general ability to solve problems - it's more just a set of very different skills that corelate to much lesser extent than people usually think and you can be really good at something that people use to judge your abilities and at the same time really bad at something else that is actually required to get the job done.
I studied sociology and I shortly worked as a data analyst. It seems to me that this kind of work requires... ehm... a different intelligence than programming. You need to be good at connecting the dots, noticing things, seeing patterns. This is my kind of thinking and I have always been good at this - doubting everything, seeing pure assumptions where other people saw 'truths', permanently creating hypotheses and alternative theories, trying to spot logical fallacies in prevailing theories... basically trying to spot things.
Programing is very different (at least it seems to me - so different tat it's even difficult do describe it). I guess it's about creating stuff, not just observing stuff. You need to build very complex and abstract mental models, keep them in your head and be able to operate with them - and this is the part of intelligence that I seem to be lacking. It just does not feel natural. I try to solve some problem and I am thinking... if this condition and that condition but at the same time not that condition... and bang!, suddenly I am lost and I don't even remember what I am doing. I cannot keep it in my head. I totally get what OP was saying about passing anonymous functions in JavaScript - I had the same experience. The first time I encountered something so simple as JavaScript closures it took me two hours to get it. And the day after that I had to repeat the whole mental process to get there again because I somehow lost it over night. This is simply not how my brain works and I think I am really bad at this. Yet people pay me for this... which is just depressing (and you understand why I write this under throwaway).
I remember our 'statistical analysis 101' professor always telling us 'remember, you are not really testing hypotheses, you are only testing indicators!' - if HR department picks wrong indicators for the skills that they actually need both company and employee are going to be unhappy - and I think this is very common because our understanding of the indicators for different kinds of 'being smart' is still poor.
The first time I encountered something so simple as JavaScript closures it took me two hours to get it.
Closures are not simple. They are a powerful concept, one which introduces a whole new way of thinking about programming compared to what most people start out doing. It's normal for it to take a long time, over many sessions, for the concept to sink in and become a part of your programming vocabulary.
Freshmen year of undergrad, it took me well over a week to wrap my head around this "public" and "private" concept of classes. I had only written C++ that was mostly C with structs and C++ input/output. I had never had to reason in an object-oriented way before, and it was different. It was completely alien to me that I would intentionally "hide" some parts of my code from myself. Now, it's completely natural to me.
What the author did not see was those same developers learn about closures for the first time. When she sat down with them, they had already reasoned with the concept many times. Don't confuse initial difficulty with some inherent deficiency in your mental ability.
I've been thinking about the same problem for a while, and I'm also a self-taught developer that looks way towards sociology. I have a little different spin on the main hurdles of doing programming. With proper tools and the right mindset, the entire goal of the organization process is that you DON'T have to think and hold 100 things in memory at once. It works okay for some problems, but any problem with a sufficient complexity (like, say a full rich website with lots and lots of features) will be nearly impossible to work with if everything was intertwined.
I find the hardest park of programming is getting over hurdles as they come up, over and over and over again. It takes a while to get over fear of small failure. Also, if you're constantly reanalyzing and have a perfectionism streak you can be paralyzed quite frequently. Have you run into the same problems?
Exactly! I would say that complexity is the biggest problem form me. It makes it impossible to keep the whole mental model in my memory and I need to reanalyze and rebuild it constantly depending on the part of the problem that I am trying to solve right at the moment.
> * I get by because everybody thinks I am smart but I feel like an impostor*
That's a real thing. Almost all of the wildly talented people in my life feel Impostor Syndrome[0] at some level. Three Panel Soul even did a spot-on comic about it [1]
You're conflating intelligence and knowledge. Intelligence is a matter of capacity and capability; knowledge is one of the things intelligent people have a great capacity for attaining and retaining. The main thing, in fact.
Knowledge, unlike intelligence, is in practice infinite. And now you can understand the inverse Dunning-Kruger effect: intelligent people have the capacity to recognize how little knowledge they have, and can ever have, about any given topic.
I don't think I am. I guess that you are probably referring to that stuff about JavaScript because that is the only part of my post where I mention something knowledge-related. But I am not talking about the lack of knowledge of JavaScript closures or functions passed as arguments - I was talking about the fact that it was surprisingly difficult for me to grasp that concept while trying to acquire that knowledge. Let me use your own words - it was 'a matter of capacity and capability'.
If you think I am wrong please explain how I am conflating intelligence and knowledge.
Oh I agree, I am just saying they miss both the part about how GMA is best when paired with a work sample and that those puzzles are not necessarily good equivalents to an IQ test.
> Spoiler alert: to solve this problem, you need to know how to enumerate the rationals.
This problem was addressed nicely in this functional pearl by Jeremy Gibbons, et al.: http://www.cs.ox.ac.uk/jeremy.gibbons/publications/rationals... . As interesting as the result is, however, it's a pretty well-made point that research-level ideas from the programming languages community are not really software engineering interview material in the vast majority of cases.
This is yet another example of "rockstar developer"-itis, wherein startups are given to believe that they need the best of the best when in fact they do not. This particular example is entirely egregious because they asked her about something that requires enumerating the rationals when what they really wanted was an iOS code monkey. Then they fired her, based on their own shoddy interview.
The worst part of it is that it's easy to feel like it's your own fault when something like this happens. The real problem was the hiring manager misleading the applicant about what the job would be, but the OP clearly felt responsible for it.
I had a similar experience (working for family, at that), where I joined their R&D group and it went well for a while, but eventually they decided they wanted to get into selling solar power and didn't need a software/electrical engineer, so I was lectured about this or that bullshit and eventually brought into a meeting with the IT director and offered a position on his team. I realize now that they were trying to make me feel unwelcome, but I felt entirely responsible at the time. I never felt like I had a choice in taking the IT position (or lose my employment there entirely), and I hated it. The new boss was, for the most part, a nice guy, but he turned into a major nitpicking asshole over little things.
I'm at a new place now and it's still hard to kick the habit of trying to avoid coworkers because you know they don't like you and want you gone.
Culture has a huge impact on hiring and working. I wish I had been taught about culture when I was in school, I could have avoided the problems I faced had I realized the warning signs earlier.
The need the best of the best but they are identifying the best the wrong way. Maybe they need the best CRUD developer, well if all you look at are candidates with strong research-level knowledge then they will not be the best at building CRUD apps. Sure we can all say that the research-level code is harder and naturally that makes that candidate smarter, but it's like trying to find the best doctor for a nursing position.
I actually like asking math questions on interviews. It shows how people approach a problem. Asking code questions in an arbitrary interview setting shows just about nothing - no access to a reference doc, somebody peering over your shoulder. Heck, I couldn't code my way out of a wet paperback in that setting.
Certainly, asking only math questions is stupid as well, people should know at least a little about the stuff they're supposed to work with, but teaching an actual language to a smart person eager to learn is a breeze compared to teaching problem solving to someone who memorized the reference manual.
Asking math questions might exclude some good candidates. I know more than a couple very productive programmers that did not go through a formal CS education. Asking math questions to those candidates could even scare them for no reason. I hired people who had no idea about Lagrange Multipliers but were able to ship code in various languages and even learn new paradigms when necessary.
There are not only smart people and persons referencing reference manuals. Being a programmer often means solving bugs in messed-up codebases, build web apps using the technology du jour, or making data go from one place to another, and asking math questions does not help a lot to find people able to do this. This blog post resonates with some people I met.
I have been programming for a while, and went through a CS education, but my experience with hiring made me realize that being good with maths and being a productive programmer are not necessarily two things that always come together.
> Asking math questions might exclude some good candidates.
Every question can exclude a good candidate, especially if you only ask the question and tick the "correct/incorrect answer" box. However, often you can learn the most interesting trait from asking a question which the candidate can't answer right off the bat: How does the candidate deal with failure or lack of knowledge. Does she/he start guessing? Does she/he ask the right questions moving in the general direction of solving the problem?
I'm not checking for academic knowledge in interviews.
> and even learn new paradigms when necessary.
This often requires knowledge about the stuff you don't know. That is a value of formal education: Not the stuff that you memorize, the bigger value I derive from my formal education is all that "I know that there's a solution to the problem but can't remember exactly" kind of knowledge. I can't remember all the sort algorithms I had to code, but I remember there's more than one and that there are tradeoffs between all of them. So if I'm constrained on memory and have a pre-sorted list I can go luck up how bubblesort is implemented exactly. That's a knowledge that self-taught programmers often lack [1].
> being good with maths and being a productive programmer are not necessarily two things that always come together.
No, nobody proclaimed so. But having a trait for problem solving and logical puzzles certainly helps :)
[1] n.b: often. Some of them have read and digested tons of theory books which could count as formal education.
> However, often you can learn the most interesting trait from asking a question which the candidate can't answer right off the bat: How does the candidate deal with failure or lack of knowledge.
Some research suggests that tests are much better at predicting performance than informal grading.
I agree - whenever interview processes come up, commenters mostly criticise the interview processes for excluding good candidates.
But that's only one part of the equation - the number of unsuitable candidates that slip through is normally more important.
Suppose that somehow we magically know that 20% of candidates would be good hires - and the other 80% are unsuitable. But we don't know which are which!
As an interviewer, I'd be very happy with an interview process that discards 50% of the good candidates and 99% of the unsuitable candidates, because that leaves me with 10 good hires and 1 unsuitable hire for every 100 candidates.
On the other hand, if a different interview process discarded 20% of the good candidates and 80% of the unsuitable candidates, that would result in 16 good hires and 16 unsuitable hires - which would be disastrous!
Even from the point of the interviewee, one probably wouldn't want to work somewhere where 50% of your colleagues are not suited to their jobs!
Summary - it's a shame to discard good candidates, but it's worse to let too many unsuitable candidates slip through ...
Hell, even those can exclude a good candidate, especially at larger companies. Candidate not a good fit for the team? Probably won't do as well as one who is.
I think you don't get the point. Learning a new language after you programmed for 5 years in a variety of paradigms like assembler, object-oriented, purely functional, will take you between a couple of hours and a week. But if you have NOT had those 5 years of varied programming experience, then the new language is not your problem. Learning the concepts is, and that will take time.
I think you don't get my point: I expect a domain expert to be a domain expert, but I don't expect a trainee to know anything and I'll structure my interview accordingly. I'll ask the domain expert questions about his domain. But I don't expect a domain expert to know the programming language that we work with just because he's a domain expert. And seriously, I'll prefer the eager-to-learn math graduate with little coding experience over the bored cs graduate that knows the java reference doc by heart and now thinks he knows how to build stuff. In any case coding examples in your interview will show you little beyond the fact that somebody memorized the docs.
The actual pain point I read from the article is a different one: There's a mismatch between hiring process and expectations. If I need a lead programmer for the iOS project that I'm about to start next week then I can't hire anyone that doesn't know Objective C and then I absolutely need to structure my interview accordingly. But if I have a couple of weeks more I'll happily teach him. And if I'm hiring somewhat smart I try to avoid those "we need someone urgently" situations.
I think you vastly underestimate the time it takes to learn these sorts of things. Sure, you can teach someone to write a simple, passable iOS app that does a few basic things in a couple weeks. But they're still going to be a raw-novice iOS developer. Maybe the app you need them to build is super simple, but if not, you're doing yourself and your company a disservice by not hiring someone who's done iOS before.
I'm speaking from experience here: I learned iOS (even after having previous MacOS X desktop devel experience) on the job, when a friend asked me to write an iOS app for her startup. I learned quickly, but made a lot of mistakes in how I structured the app that came back to bite me months later. If I'd had the time to start over from scratch, I would have done things quite differently and the whole thing would have been a lot easier.
And I was slow. Every new framework I had to learn slowed me down and added days to implementing the part of the app that needed it. A seasoned iOS developer wouldn't have run into problems like that.
I think you vastly underestimate the value of "other knowledge" for any kind of complex undertaking. Unless you get a domain expert who happens to be a rock star programmer and a ninja architect and top-notch devops, you'll have to compromise somewhere. Take an iOS game: Depending on the game it can be vastly more value to have knowledge of game programming and game AI than having knowledge about iOS frameworks. If there's a team on the project I'd even go as far and say that I would pick the domain expert over the framework expert as team lead, given that all other capabilities were on par.
Sure, there are always compromises. I wasn't claiming otherwise. I was merely pointing out that ramping up on a particular platform is nowhere near as easy as the parent suggests.
Games are a bit of a special breed, though, and there are several game engines out there that hide the platform pretty well. I draw a bit of a distinction between "building and iOS app" and "building an OpenGL app that runs on iOS". The latter is much easier if you lack iOS experience but have the required OpenGL skills.
I also think that there might be a mismatch between hiring process and expectations. When you say "But if I have a couple of weeks more" then you obviously underestimate the time it takes to learn if we are actually talking about somebody without much programming experience. Try "if I have a couple of years more".
See, there's so much more that you need to learn when you start on a project in a company. There's the framework they use, their libraries, the requirements for the project, the problem domain, the people you work with, the process they use, ... just to list the few that come to the top my head. Especially learning the problem domain can be very hard and challenging to programmers with little or no knowledge of the field. It can easily take a couple of month, if not years to become reasonably proficient.
So yes, "a couple of weeks" is certainly not enough to transform a graduate into a project lead for bespoke iOS project, but it could easily be enough to transform a decent java programmer with solid project lead experience. Transforming a crack Objective C Programmer with no lead experience can take easily as much time, if not more.
My gist is that programming knowledge will only get you so far and depending on what position you're hiring for other experience combined with the ability and willingness to learn may be much more interesting. And whatever I do, I tend to hire for that trait since that allows people to pick up other abilities when required.
Learning a new language is fairly easy. Learning the language's standard library to the point that you don't have to look at the reference docs every 30 seconds takes a bit more time.
The author clearly showcases how this is not true in his (and probably others') case. Anonymous functions, function pointers, etc. are not language-specific features, but rather something you learn when learning to program.
I think the point of the article is that being smart at puzzles is not enough. It may not matter whether you have 3 or 10 years of programming experience, but if you have 20 years of puzzle solving experience with only 6 months of (real-life) programmering experience you have a lot of non-trivial learning left to do.
Being in a startup setting where everyone (usually) is expected to be a bit of jack-of-all-trades, I would argue that having programming experience trumphs being smart, at least for the first period. You may be the master of finding smart algorithms to design your application, but if you can't build the first CRUD website it does you little good.
So true. I've been a professional developer for 15 years and I've worked with several languages (C++/Java/PHP/Python/Perl/C#/Ruby/JS/Groovy), but man, does this whole "functional" stuff is being hard to get a hold of.
Specially Scala, for me, is very hard to understand the syntax, even after a year of working with Play. The most frustrating part is understanding the flow of the data in chained calls like in map-reduce.
Did you read the OP? The point of the article was that one can be great at that sort of thing and yet quite terrible as a production programmer. You are not measuring the skill you want with math problems, you are measuring a proxy.
Furthermore, I've known plenty of smart math people that just never seemed to be able to program (well). I think they are different skill sets, certainly with plenty of overlap, but plenty of differences as well, and those differences matter. I laughed out loud at the "variables, variables everywhere!" answer in the OP - I've had to deal with so much code written that way. Some people, very smart people, just don't 'get' design in that way. I worked with a guy that used to run around the office, asking brain teasers, sharing tidbits of knowledge, but he couldn't execute a basic project - couldn't plan what to do, couldn't do things in a rational order, couldn't experiment and gather data, couldn't incrementally design, develop, refactor, nor do a big-bang waterfall kind of design, and so on. He was not sorely missed when let go. Smart as a whip, and useless (for programming).
I've worked at several companies with staff mathematicians. Sooner or later they got their hands on a compiler. Oh my. No, let me do that. You tell me what is wrong with my Kalman filter, but I'll take care of the implementation, thank you.
Its easy to kvetch at somebody else's answer without offering an alternative (I do agree whiteboard progamming is disastrous). So, instead of asking math, why not ask them to write a simple routine, but then start asking real world problems about the code they would face - how would you make this API interface robust? What kind of documentation would you write. How would you handle errors? Is this code exception safe? Thread safe? How would you make it either/both of those. Suppose your problem size was n=100MM, how might you need to change this (say they have a data structure that loaded everything into memory)? Ask them some problems they will see in production - what is the network delay, or whatever your problem case is. You still get to see how they approach problems, but in the context of the actual decisions they will be making while programming for you.
Anyway, that is what I try to do. I am revising my thoughts even on that, because I find people flopping on the 'code the simple problem (and, it is simple)' yet doing great on all the engineering questions, and doing fine if we hire them.
> Certainly, asking only math questions is stupid as well
So what I actually advocate is asking questions about the field that the person is going to work in. I still like to ask some math questions - a lot of the stuff that programs deal with basically fall back to math. Heck, all relational database stuff falls back to set theory, so knowledge in that area certainly helps a lot. I'm not advocating employing pure theoretical mathematicians to develop contact forms in php. However, from my experience the following two statements hold true:
* Studying CS does just about nothing good to your actual programming skills and your knowledge of real-world problems. So just don't expect recent graduates to be able to develop a program in a rational order, using a reasonable process, refactor or any of the skills you're asking for just by virtue of having "programmed". If you're looking for an experienced programmer hire for an experienced programmer, but the OPs tale clearly shows that she was no experienced programmer - why would I quiz her in the way I'd quiz a 10-years-learned ruby programmer?
* If you have time to teach and educate a programmer, prefer a smart and eager type over someone who spools down memorized knowledge. While memorized knowledge can sometimes help, the ability to learn and educate yourself is much more helpful once your memorized stuff is not saving you.
Yes, I did read your post. The point is whether asking math questions at all is valuable compared to the other questions you asked. I do not believe you particularly addressed that in your post or in this one. It is a proxy for what you are actually trying to measure - why not measure what you want to measure?
Your first asterisked point does not at all square with my experience. It depends on the program of study; I've certainly come across kids with no real pragmatic ability, and other programs that turned out, as much as you can in that environment, very pragmatic and skilled programmers. Certainly this is a skill set that improves over time. But let's not quibble over that; you raise a larger, valid point about interviewing recent grads vs more experienced people.
As far as that goes, I try to question them about school projects. "So, if I was to try to take that and do Y with it, what would be the consequences" type questions. Like you w/ the math questions, I do not expect any particular expertise and experience in actually solving the problem. But, I can start to see how they think about things. If they aren't thinking that clearly, throw them a bone and see what they do with it. Does it give them an 'aha' moment that then leads them to a better answer (I infer, perhaps incorrectly, that they can learn and be mentored), or do they just stonewall, not make the connection, or what have you. My suggestion is pretty simple. Measure what you want measured, not some proxy. I will point out that recent data suggests I am right. Google has admitted that all of their algorithm type questions are not good predictors of on the job performance, but questions about experience and "how would you X" are. I don't consider their data the last word on the subject (their hiring is quite narrow after all), but certainly suggestive.
I absolutely agree with asterisk 2, so I didn't address it.
> The point is whether asking math questions at all is valuable compared to the other questions you asked.
The author uses a graph search problem as an example - which is a very typical problem in IT. How do you approach such a problem? This is a valid math question that certainly adds value compared to other questions I might ask. It actually checks for multiple things: Does the candidate have a grasp of the underlying mathematical concepts. How does the candidate approach a problem decoupled of the actual real-world constraints of a programing language? Can the candidate describe a problem in clear, concise terms? Same for set theory: intersection, union, functions mapping input to output. Complexity analysis of algorithms - all of that is valuable knowledge and it's absolutely valid to ask people for that. How does binary AND/OR/XOR work? How does an exponential decay curve look like compared to a gauss or linear curve? This GH issue https://github.com/elasticsearch/elasticsearch/issues/3423 was posted on HN as example of a great feature description and it's full of formulas describing how it works. Vectors, matrix multiplication is a fundamental thing when you're doing natural language processing. Statistical problems are not exactly uncommon in programming either. Map/reduce are mathematical concept. What's wrong asking for that? Failing the answer does not mean that the interview is over, but to assert a candidate I need to know what she knows. As annoying as it sometimes is, math is _the_ _fundamental_ underpinning of what we do every day.
> As far as that goes, I try to question them about school projects.
Totally fine and I agree. Still - what's wrong with asking math questions?
> Google has admitted that all of their algorithm type questions are not good predictors of on the job performance
You're running into a problem if you hire only for math knowledge. But -repeat- that's not what I've been advocating.
What kind of math questions do you ask? Are then relevant to the job? Is it knowledge that the employee will need to have for their day-to-day tasks?
Asking math questions that are unrelated to the employee's tasks has strong bias for recent graduates. Ask a 40 year old a question about an equation they haven't used since they were 19 and they won't remember.
I was once asked to do binary math on a phone interview. I hadn't had to do binary math since I was in school 10 years prior. I was pretty much guaranteed to fail. This was not a good test. If I needed it I could easily relearn binary math in a few hours at most.
Memorising the manual is the strategy of a surface learner. Surface learners can make pretty good PHP programmers, but they'll struggle with, say, passing Javascript closures.
A deep learner will look for the underlying principles and abstractions. They can quickly get an overview that, even when it's fuzzy, is still accurate enough for them to know where the gaps in their knowledge are, so they can fill those gaps quickly when they need to. They're like Mendeleev with his first periodic table. He didn't need somebody to show him a sample of gallium to know that it existed: he could inferred its presence and properties from the overall structure of the system.
Of course, a really advanced programmer will have done both. They will know and understand everything. But those guys are few and far between.
What I mean is that people who enjoy programming probably tend to be smart. I haven't done any statistical surveys, but the motivational feedback loop for learning programming rewards intelligence.
I doubt that. There's a lot of people going into programming because they think
a) IT is akin to playing computer games all day.
b) It's easy work in a climatized office with solid pay
c) The jobs are relatively secure and abundant.
I've seen many people go into IT that would much better have been employed elsewhere, so I think that we, as programmers are not more intelligent on average than the rest of the population. I might agree that people who enjoy programming have a knack for a certain type of intelligence and problem solving ability, but I don't think that programmers are limited to the group of people that enjoy programming.
looks like the one part you said you might agree with in your very last sentence is 100% of the group that they were talking about: people who enjoy programming.
Actually, since I'm the author of the post they're referring to I can say that the group we're referring to is "programmers" or rather "people you'd hire as a programmer" which is a superset of "people who enjoy programming." Even if you only consider people who enjoy programming I'd be very careful about calling them "more intelligent". They're probably good solvers of a certain kind of logical puzzle, but intelligence encompasses much much more than that.
*presuming that you went to an undergraduate institution where the variance in student intelligence is large enough to detect an appreciable difference between the avg intelligence in these two majors.
The phenomenon of the Computer Science graduate who can't write a FizzBuzz program, or even the post-graduate who can't write a simple recursive function, is well attested. But a Math student who can't handle a recursive definition is unlikely to make it through the first term.
> I would guess that there is actually significant overlap between these two groups.
I wouldn't. Smart people get bored sitting around memorizing things. They'd rather be thinking. Purely anecdotally, the smartest people I know rarely have encyclopedic knowledge of anything.
But thinking has the side effect of storing lots of stuff in your memory. So you can end up with encyclopedic knowledge of subjects that you've spent a lot of time thinking about. It just isn't knowledge that got stored by explicitly trying to "memorize" things (which means it's more reliable anyway, since it's knowledge that's connected to other things you know).
I store a lot of "I've read the solution for that problem somewhere while researching something else." I still can't recite the stdlib doc for any of the programming languages I work with. There's only so much stuff I can cram in my head and reference docs are fairly easy to look up.
Not necessarily; an encyclopedia isn't just an unconnected catalogue of facts--at least, it's not supposed to be. An encyclopedia article about a given subject is supposed to show the subject as a connected whole; it will contain facts, but will also contain important relationships between the facts, general principles, theories that explain the facts, etc. If you have that kind of knowledge of a subject, you don't have to memorize all its facts, because you can easily get to them from the facts you do have memorized via one of many interconnections.
After much experimentation giving interviews for server side positions, I've come to favor questions that involve routine real world problems that can be handled in increasingly sophisticated ways.
One example I use is getting the candidate to write crud, list, and search controller actions for a simple category data structure. Given a basic category data model (e.g. Name, Parent), the candidate starts with the crud actions.
Crud actions aren't meant to be difficult to solve and serve as a basic screener to verify the candidate has working knowledge of the basics. The only edge case I look for the candidate to ask about is if orphaning child nodes is allowed (I.e updating parent node, deleting a node with children)
List action(s) start getting more interesting since recursion comes into play. A basic implementation of an action that can load the tree given an arbitrary category as a starting point is expected. If the candidate has some prior experience, a discussion of what performance concerns they may have with loading the category tree is a follow up question. The tree loading algorithm is then expected to be revised to handle an optional max depth parameter. An edge case I look to be considered is how to signify in the action response that a category has one or more child nodes that weren't loaded due to a depth restriction.
The search action implementation has a degree of difficulty scaled to the candidates experience level. All candidates have to write an action that returns a collection of categories matching a search string. Those with previous experience are asked about a paging solution. Senior level candidates are asked to return matching categories in a format that indicates all ancestors ( for instance: "Category 1 -> Category 1.1 -> Category 1.1.1" result for search string "1.1.1")
For an added degree of difficulty, candidates can be asked to recommend data model tweaks and algorithms supporting tree versioning requirements necessary to allow for loading the category tree's state at a given point in time.
The candidate's performance to this exercise seems to give some insight into their level of experience and ability to implement algorithms from a common real world example without having to ask much trivia or logic problems.
And even better - knowing that quite a few database engines have extensions to help handling hierarchical data and it probably depends on the details as to which is actually better in a particular circumstance.
Definitely. Nested sets have a lot of weaknesses, and any decent modern database will have CTEs so you can just use an adjacency list with a recursive query, or materialize out the closure table.
I am not able to understand any of the questions because the jargon used is unfamiliar although the concepts look simple. Are all these terms specific to a particular framework ? Genuine question, thanks.
No, from what I see they are not. CRUD relates to "Create, Remove, Update, Delete" which is the most common set of actions when dealing with database records.
Then the GP goes into detail about a tree-like structure of categories and their parents. If you're not familiar with trees and graph algorithms this might be quite daunting.
Do some googling on the terms I mentioned and please ask about specific things you do not understand.
Hiring engineers is hard, and companies haven't really figured it out yet. Even the best companies rely on puzzles and gimmicks that often have little to do with day-to-day programming.
At one company I interviewed with, I was asked to implement a queue using two stacks. At that time in my programming career, I had worked with C, C++, Obj-C, Lua, Python, JavaScript, SQL, and a handful of DSLs developing games, game development tools, and web applications. Want to know what I had never done? Written a queue using two stacks. My immediate response to the question was, "Why would you want to do that?"
If you really want to know if someone has the capacity to pull their weight as an engineer, ask them about what they've built. Even if they are fresh out of college, the best engineers will have projects they can talk about and explain. Ask how they approached/solved specific problems. Ask what they're most proud of building. Ask what was most frustrating.
Those are the kind of questions that will provide insight into a person's problem solving capabilities and offer a decent picture of what they're capable of doing.
If you're wondering, the "two stacks queue" is an easy way to write a simple immutable queue for a functional programming language. You use one functional stack for enqueueing and one for dequeueing. When the dequeue stack is exhausted the enqueue stack is "flipped" into the dequeue. Amortized runtime is O(1) per operation.
Once they told me to use one stack for enqueueing and one for dequeueing, I immediately solved the problem. What I didn't understand is why you would a.) implement your own queue structure and b.) implement it in such a non-intuitive way.
Now that I know this would be for a functional language, it makes way more sense. At the time I couldn't get over why you wouldn't just use a doubly-linked list like the standard implementation in most OO languages.
Exactly. "Tell me about your last project" can easily turn into a 2 hour conversation with any developer who has built anything of substance. It will tell you everything you need to know about them.
I definitely agree with a model that places emphasis on past work/projects.
"To find out if they can get stuff done, I just ask what they’ve done. If someone can actually get stuff done they should have done so by now. It’s hard to be a good programmer without some previous experience and these days anyone can get some experience by starting or contributing to a free software project."
http://www.aaronsw.com/weblog/hiring
Best way to go is to ask them about their projects and hear them as to how they will explain them and how enthusiastic they are about building new things and ask them to probably build one in 2 days with the help of google. Then i think the quality would just come out naturally and now you can decide whether to hire this guy or not.
The problem is, you often want to know what they can build for you in the future. And old projects where someone worked on something very specific for 5 years and exercised a small subset of a particular language or general algorithmic tool kit may not be predictive of their ability to work on YOUR project
Here is an interesting idea that I had reading this. As a startup, what if you were to create a simple computer language that looked different from most other computer languages, at least somewhat different. Alternatively, just use one of the many really obscure programming languages out there, just make sure the applicant does not know it ahead of time. Give the applicant a 10-20 page reference manual for the language and ask them to make a simple program of some sort. Have them read the manual and write the program, hopefully while not looking over their shoulder, so they can relax. In the manual you give them omit one critical function or API reference, but make sure that info is available online (make it available if you made up the language). Then see what happens.
This would test programmers ability to learn a new language.
Hm, maybe if I ever have to hire somebody I'll challenge them to a Corewar duel :-)
Before the internet my friends and I sometimes used to get to together, everybody got some time to write a Corewar warrior on paper, and then we'd watch the tournament together. It was great fun.
You would be testing them on the ability to learn a new language. You could test the same abilities using just a custom library. This is a test I used to use on intern candidates. I didn't even provide documentation, just examples of API usages, and was testing their ability to learn from example.
I guess that proposed interview is supposed to get people out of their comfort zone and test whether how quickly they can adapt and learn from a manual and simple examples and then at least be able to write simple programs.
It is rather unfortunate how little correlation most tech interviews have with their respective jobs. It's largely a lose-lose situation for everyone. Developers who could easily build great systems but aren't experts in graph theory get passed over while brilliant mathematicians who can't necessarily code get hired. Result? Companies simultaneously having to fire employees while facing a supposed talent crunch. Given that this hurts everyone, how did we even get into this situation?
Probably because the only person who doesn't lose from this is the interviewer: they get to have fun. Honestly, when you spend all day buried in code, it's fun to play with puzzles for a change.
Perhaps it's time we started optimizing interviews for hiring success rather than interviewer happiness.
Generally speaking, high quality people won't take a short-term contracting gig. They are probably working elsewhere and don't need that level of uncertainty; they're looking for something better, not just anything.
Best people I know don't even give or take interviews.
They generally have a long history of awesome projects they have worked on, and they've built years of reputation through working on such projects.
And they just don't pick up any other project in any other company. They are pretty much clear about the best kind of problems they want to work on, and they don't wish to waste their time other than that.
Every time a company boasts about having a process to hire only 'A' candidates I chuckle. The 'A' or even 'B' candidates aren't even up for hire. The interview process begins with 'C' people.
Generally speaking, most programming position don't need to be filled by high quality people. There are lots of smart, reasonably competent programmers out there, but who don't have years of experience on their CV, whom are eager to learn and would happily take a day or two off for a chance to work at an interesting company.
Truly high quality people are generally known by their reputation and are headhunted rather than expected to answer job ads.
I'd hope that by that point I'd have sold them on my company enough that they really thought it was better.
Also, I don't think it needs to be a long gig. Maybe even just a day, which isn't much longer than the gauntlet of technical interviews some companies will put you through. Even that short period should be enough to see if someone works well.
Doing a day's contracting elsewhere would violate my current employment contract. (And I think strict adherence to the letter of the rules will be reasonably common among good developers)
Sounds good except for every job opening, I get 40 resumes, interview 15 people and of those 5 look reasonable. Which of those 5 do I give the short term contract gig to? So you're back to the same problem.
> Which of those 5 do I give the short term contract gig to? So you're back to the same problem.
Ideally, all of them. It's not hard to find a couple one-off tasks that you'd like to see done in a day. Or could could proceed sequentially until you find one whose work you like.
Funny. Just made a hire and this story made me think of it.
The position I was filling is a part-time position for a CS major, sort of like an internship. I devote time to develop his/her skills, s/he would get real-world experience, and a little money to help with cost of living. If everything works out, a position could open up for full employment.
I had a pretty good idea what I was looking for. Someone that had good grasp on theory but had no experience coding. Preferably enrolled in Uni. I had 5 applicants but the only candidate I interviewed is enrolled in Math-CS.
I basically tried to gauge if he had deep interests and asked him to code a bit, solve a simple control (find me the article with the highest hitcount from the day a week ago, gave him 10 minutes).
He failed the coding test but I made the hire regardless. Reason why was 2 things out of the 4 hours we spent together: When I asked him who he considered the father of CS he rattled off von Neuman, Djikstra and Knuth. Yeah, you can make that argument I suppose, but he knew who the influential people were. The other thing was: even if he failed the coding test he failed it by not reading the code examples quite right, he was using my code to try to help himself solve the problem. I'm sure he'll work out.
We as a field should employ internships a lot more than we do, get the college kids and undergrads working on real-world problems a lot more than we do.
I agree. But for a different reason: I'm shit at maths puzzles.
I just don't have the experience or tools or interest for them.
And yet, somehow, in 20 years of business geekery I've never come across a problem I can't solve.
Maybe when writing Tetris for J2ME I would have saved myself 10 minutes googling if I'd had the experience to realise that right angle based matrix translations don't require fp maths and maybe when writing financial indicators, I'd have saved myself half a day if I hadn't had to look up integrals but this sort of stuff is definitely in the minority as far as my experience goes.
I don't play Sudoku, I don't solve puzzles. I don't want to solve mathematical puzzles. I just want to deal with system and deal with security and none of them requires me to understand the tricks to solve a puzzle. Sure I can learn some cool algorithms but no thanks. I don't want to solve puzzles. Exactly.
I believe this is deeply valuable. For some roles, I would much prefer to hire someone who can quickly see the value of breadth-first search from both ends.
If he/she doesn't happen to know the syntax of Ruby, or Java, etc. it's less important to me.
This is your company, correct? (I stalked your Github profile)
Seems like the types of problems you're solving are exactly those which require far more domain experience with Ruby/HTML5/Javascript/whatever than the ability to see the value of various graph-searching techniques.
Would you hire this guy even though he's said quite plainly that he lacks the experience with these technologies (and has difficulty picking up new ones owing to that lack of experience)?
Some of our projects have us on site, side-by-side with a client's staff programmers. These projects involve millions of dollars, hundreds of stakeholders, and years of existing code. The programmers have a wide range of skill levels.
The work in these projects involves figuring out the project's objectives, goal decomposition, some agile/lean PoCs, then developing the BDD, MVC, DCI, API, CQRS, SQA, etc. Much of this can happen in pseudocode.
We also do pair programming, code reviews, brown bag demos, cross-training, and the like. I believe all these can help with developers getting up to speed with language syntax.
That said, choose the right person for the job. YMMV.
I agree as such. Given the the choice between someone who's really smart and can solve hard problems, but a mediocre programmer or someone who's a good programmer, but sucks at problem solving, I'd chose the first one. That being said, if you hire someone in the first category, don't expect them to be happy and competent at writing an iPhone CRUD app in two weeks after seeing Objective-C for the first time.
I think that was the thrust of the article. If you're hiring based solely on someones math and algorithm skills with zero concern about their coding skills, you cannot turn around and be angry when their coding skills aren't what you needed. It's not math vs programming as such, but more generally about tuning your interviews to finding the skills you actually need (as opposed the skills you think you need).
I think you're still suffering from same mindset that led to the problem described in the article though. He wasn't a mediocre programmer - he was a bad programmer. If you make a lot of assumptions about people in an interview you will probably be disappointed.
This is great. Now what would you do if after three months, your new hire can still not ship code that benefits your business? Fire the person? Keep teaching?
I also believe algorithmic knowledge is important, and tend to give algorithm questions to my candidates, but it matters more for those who will write databases and game engines than for those who will write CRUD apps.
I was good at the algorithms and data structures part of my university course, but as a developer with ten years experience, I find it is very rare to actually use these now.
Far more important is how / when to index an database table, how to design the tables in the first place.
If I need an efficient algorithm, in most cases someone has written (and tested) it and put it in a library.
Bringing it full circle, a puzzle is just a peculiar form of algorithm, most of the time. You don't want to hire someone who will sit around thinking about graph puzzles and how to measure 4 liters when you only have a 5L and 3L beaker. Google it and get on with the job.
If the person wrote the code in such a way that no one else could understand that it was a breath-first search, is that valuable? If they didn't even leave comments when they write code? If they didn't realise that you're using some slightly arcane feature of your language to keep a library of useful tree operations, and was expected to use the pre-existing function through macros/generics? Or was expected never to use such a technique because half the team wouldn't understand the code?
Hey, I know the theory on how to bake a large amount of cakes, breads, cookies, etc. I learned it by watching it on TV, and from cookery books. But actually it turns out that I'm a pretty crap baker, because I lack experience to know when I'm over-working/under-working dough, how to adjust cooking time with an unknown oven, etc. And I make a huge mess when I'm doing this, and take a lot longer than anyone with basically any experience of doing it at all.
I'm dealing with this now, having been interviewing for different engineering roles over the past two months. It hasn't been as bad as straight-up conceptual math problems, but there have been plenty of questions that I have questioned for validity.
Interviewer: "How can we optimize the character replacement in a string such that we use no extra memory?"
Me: "We do this and that and this. But, should we consider what situations we would need this optimization?"
Interviewer: "What? Why?"
I can now use this as a filter as I interview organizations. Optimizing algorithms by creating your own core data structure classes (instead of using the built-in ones) is great in certain circumstances, but an absolute waste of time in many others. And if you're not going to ask me about those times when making those improvements is important, then you're not asking questions for a programmer -- you're asking them for a theoretician who can recall syntax.
There are many discussions here on HN about company hiring procedures. Company hiring procedures and their effectiveness is a heavily researched topic, but most hiring managers and most job applicants haven't looked up much of the research. After reading the blog post kindly submitted here and some of its comments, and then reading most of the comments here on HN that came in while I was asleep in my time zone, it looks like it's time to recycle some electrons from a FAQ I'm building about company hiring procedures.
The review article by Frank L. Schmidt and John E. Hunter, "The Validity and Utility of Selection Models in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings,"[1] Psychological Bulletin, Vol. 124, No. 2, 262-274 sums up, current to 1998, a meta-analysis of much of the huge peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring procedures. There are many kinds of hiring criteria, such as in-person interviews, telephone interviews, resume reviews for job experience, checks for academic credentials, personality tests, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.[2]
EXECUTIVE SUMMARY: If you are hiring for any kind of job in the United States, with its legal rules about hiring, prefer a work-sample test as your hiring procedure. If you are hiring in most other parts of the world, use a work-sample test in combination with a general mental ability test.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well. One is a general mental ability (GMA) test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. (But the calculated validity of each of the two best kinds of procedures, standing alone, is only 0.54 for work sample tests and 0.51 for general mental ability tests.) Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general mental ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both miss some good performers on the job, and select some bad performers on the job), but both are better than any other single-factor hiring procedure that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
Because of a Supreme Court decision in the United States (the decision does not apply in other countries, which have different statutes about employment), it is legally risky to give job applicants general mental ability tests such as a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of hiring procedures. The Griggs v. Duke Power, 401 U.S. 424 (1971) case[3] interpreted a federal statute about employment discrimination and held that a general intelligence test used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring procedure had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring procedure could be challenged as illegal if it can be shown to have disparate impact on some job applicants. A company defending a brain-teaser test for hiring would have to defend it by showing it is supported by a validation study demonstrating that the test is related to successful performance on the job. Such validation studies can be quite expensive. (Companies outside the United States are regulated by different laws. One other big difference between the United States and other countries is the relative ease with which workers may be fired in the United States, allowing companies to correct hiring mistakes by terminating the employment of the workers they hired mistakenly. The more legal protections a worker has from being fired, the more reluctant companies will be about hiring in the first place.)
The social background to the legal environment in the United States is explained in various books about hiring procedures,[4] and some of the social background appears to be changing in the most recent few decades, with the prospect for further changes.[5]
Previous discussion on HN pointed out that the Schmidt & Hunter (1998) article showed that multi-factor procedures work better than single-factor procedures, a summary of that article we can find in the current professional literature, for example "Reasons for being selective when choosing personnel selection procedures"[6] (2010) by Cornelius J. König, Ute-Christine Klehe, Matthias Berchtold, and Martin Kleinmann:
"Choosing personnel selection procedures could be so simple: Grab your copy of Schmidt and Hunter (1998) and read their Table 1 (again). This should remind you to use a general mental ability (GMA) test in combination with an integrity test, a structured interview, a work sample test, and/or a conscientiousness measure."
But the 2010 article notes, looking at actual practice of companies around the world, "However, this idea does not seem to capture what is actually happening in organizations, as practitioners worldwide often use procedures with low predictive validity and regularly ignore procedures that are more valid (e.g., Di Milia, 2004; Lievens & De Paepe, 2004; Ryan, McFarland, Baron, & Page, 1999; Scholarios & Lockyer, 1999; Schuler, Hell, Trapmann, Schaar, & Boramir, 2007; Taylor, Keelty, & McDonnell, 2002). For example, the highly valid work sample tests are hardly used in the US, and the potentially rather useless procedure of graphology (Dean, 1992; Neter & Ben-Shakhar, 1989) is applied somewhere between occasionally and often in France (Ryan et al., 1999). In Germany, the use of GMA tests is reported to be low and to be decreasing (i.e., only 30% of the companies surveyed by Schuler et al., 2007, now use them)."
It seems awfully rude to cut and paste this gigantic comment, which is now virtually pinned to the top of a discussion thread of a blog post to which it barely responds.
There's nothing in this that indicates it is at all customized for the article it is attached to. Especially given that you and the author seem to have some things in common (connection to small-town Minnesota & love of mathematics) it seems like you could have done a lot better than just re-posting virtually the same comment for the 11th time.
You have to go about half way down the page to find a comment that actually engages with the article. The way the HN system works out in practice the "winning" top level comment determines the entire structure of the ensuing discussion. I think that means that top level commentators have a particular responsibility to comment well.
The currency of upvotes suggests that many readers have appreciated my comment, perhaps because it provided them new information about an important set of issues. That set of issues has been discussed in many of the comments here, and was discussed in the original article.
The currency of actual cash money, which is not provided in our information-sharing environment here on Hacker News, would free up time for me to customize each comment I post more thoroughly. Under current conditions on Hacker News, I will continue to post for free, and I hope some of my comments are worth more than I bill for them.
A comment of this size that's not specific to the article might be better hosted on a blog and incorporated by reference, or a link to the previous discussion on HN. It's great material, but I have to agree with the GP that you post substantially the same comment every few months. It does rather swamp other contributions while being somewhat generic.
Quite the glib and misanthropic response, not to mention the value you measure is based upon imagination and a strawman. If you're not interested in engaging the community except in terms of price, why participate?
Most of the time one's hopes are not as valuable outside that person's head as they are inside.
If you're not interested in engaging the community except in terms of price, why participate?
I was interested in sharing information, did, was evidently appreciated for sharing the information, but also still criticized for how I did so. Fine. But I will go on sharing information here in a manner that fits my work schedule and other personal responsibilities. If all I dislike about somebody's comment is the MANNER in which it was written, when the content is accurate and helpful, my own style is not to complain about that.
I really don't see how this is relevant to this post, since the research you're citing is about hiring people for routinized or semi-routinized labor, like at a car manufacturing plant.
Your question about relevance is fair. The author of the blog post kindly submitted here complains that she [I wrote "he" before edit prompted by kind reply below] gets math quiz questions when applying for programming jobs. That's a mistake on the part of the hiring companies in two ways.
1) The main thing to look for when hiring programmers is programming work samples. In general, for any kind of work, a work-sample test will tell you more relevant information about which person to hire than any other kind of hiring procedure. (Some other kinds of hiring procedures have incremental validity in addition to work-sample tests, but none have higher validity than work-sample tests.)
2) Insofar as a math quiz question purports to figure out which applicants are smart and which are not, it is legally risky, as are IQ tests and AS ARE EDUCATION CREDENTIAL REQUIREMENTS if the quiz question is not validated as related to bona-fide job requirements. The company that publishes the Wonderlic personnel test hires a lot of psychometricians and lawyers to be able to defend the use of that test for hiring for dozens of kinds of jobs, and I think the balance of the evidence across hundreds of studies on many different job categories in multiple countries shows that general mental ability matters for performing most kinds of jobs well. But in the United States, a smartness test that a hiring manager just makes up is legally risky. It's a good idea to hire smarter rather than dumber job applicants, but it's an especially good idea (for legal protection in the United States) to make sure that the smartness test you use has been validated for your workplace.
The research I'm referring to is more generalizable across more occupations than you suggest. In other words, the usual relevance of my FAQ about company hiring procedures in the frequent HN threads about company hiring procedures is that companies almost always go wrong by 1) NOT using work-sample tests, a very good way to find competent workers, and 2) often go wrong by posing trick question tests, presumably to find smart people, without validating those tests and ensuring that those tests don't have disparate impact on applicants. Companies can and should do better, and research tells them how. The people who often complain on HN about company hiring procedures are largely correct, but usually just share anecdotes about what they would do differently if they were hiring, or what they would like hiring managers to do the next time they apply for a job. Research on this topic is abundant, and it applies to people seeking work as programmers as well as to most other occupations.
Thanks for pointing that out, for English pronoun usage at the very least. My eyes blazed right past the blog heading that uses (as I suppose) the author's given name. I didn't see any other author information on the post or on the about page for the blog.
I read the domain ("count aleph") and associated "count" with male nobility -- when in reality, now that I look more carefully, it must instead be a very clever pun on math, which makes lots of sense given that she (the author) likes math puzzles.
In the UK, it's illegal to block people from getting the job for anything which doesn't fall under the General Operating Requirements (GOR) of the job. This means that you can refuse to hire someone in a wheelchair for a bricklaying job (for the sake of pedantry, lets assume they cannot physically do the job) but you cannot just because someone has a certain religion.
It's a shockingly sensible piece of legislation.
Some interesting consequences that I know have been tested in court:
You can't be 'over-qualified' for a job, that's a form of age discrimination. An over-qualified person shouldn't be less able to do the job.
You can fire someone for wearing a niqab if it stops them from doing their job (e.g. teaching English to young students to whom English is a second language).
So, if you can show that not having the qualification doesn't disadvantage you then I would expect that'd be a legal issue.
Did you just say that education requirements are legally risky?
Yes, I did say that. I was surprised to find, upon a close reading of the Griggs v. Duke Power, 401 U.S. 424 (1971) case,[1] that the Supreme Court held in that case that not only was an IQ test an illegal hiring requirement, in the circumstances of that case, but so was a high school diploma requirement. The usual headline description of that case mentions only the holding about the IQ tests, not the holding about school credentials. Either kind of requirement could be challenged by a rejected job applicant based on that Supreme Court precedent.
The Duke Power decision was unconstitutional, and it involved total racial segregation and galling defiance of the courts. It has very little applicability to highly-educated employees, if for no other reason than there are virtually no Africans capable of doing the jobs.
Uh, why the downvotes? There are virtually no such Africans. It's an established fact, you just go around and take a census. Plenty of farm boys from Idaho, Japanese, girl engineers from rural China, guys from India with accents thick enough to stop bullets, you name it. All are welcome in the universities, all succeed and prosper. Just no Africans. You don't have to worry about illegally performance testing them during a job interview, because it is a fact that there are virtually none.
And if you still disagree, don't downvote to express your mindless politics. Simply give me the names of the vast horde of African engineers and scientists waiting for a chance. I will hire them and become the world's first trillionaire.
But anyway back when I was living in France I've had the chance of studying and then working with a lot of great engineers (electrical engineers mostly) coming from Africa: Marocco, Tunisia, Togo, Cameroon, Rwanda, etc.
This is neither a rare nor surprising occurrence: engineers coming from african countries are plentiful and very competent. Maybe you didn't find them because they are not waiting for you to give them a chance: turns out they are as hard and expensive to hire as anyone else...
If I was hiring, I would hire homosexual Martian Nazis if they made good employees. The few African colleagues I have had were skilled and pleasant. The trouble is that, at least in the U.S., that they are very rare.
I suspect that a lot of what you were seeing in France was selection bias. The most talented third-worlders immigrate at a very high rate to the former imperial states. France gets the Africans because they colonized Africa extensively and were not especially vicious. If you moved your company to Rwanda, you would find that very very few locals would be trainable for engineering work.
They're only risky to the point where you can't make a case for them. In the Duke Power case, there was little reason whatsoever to require a high school diploma. Using education requirements for exempt job with more autonomous decision-making and reliance on knowledge wouldn't be a problem.
After reading the OP I immediately thought of your standard hiring advice and wondered if the folks asking math questions aren't on the right track after all. (Thank you for your persistence posting it, btw.) At least half of my daily work requires learning new things on-the-fly, and I bet in 5 years most of my technical knowledge is turned over anyway--except the math and CS fundamentals. In programming aptitude matters way more than current knowledge. I think IBM discovered the same thing in the 60s when they were hiring chess players and crossworders. So if math puzzles serve as a legal(ish) proxy for an IQ test, aren't they just what interviewers should ask? Of course I'd rather have someone who knows Ajax and 3NF if I can get it, and not lose any time to up-front training, but that may not be realistic, and those things can be learned if the person is smart. Particularly for an entry-level hire, I'd rather hire someone who can solve graph theory problems than knows the command-line options for git.
Math questions are probably an okay substitute for "general IQ" testing for the majority of population, moreso than most other question categories (math is literally just logical reasoning). But, it's not a truly general skill, it's a specialized discipline--and that means the practice of using it as a screen is going to fail hard when you try to apply it to someone with specialized training. Which is what happened in the article.
I can confirm this works in my own experience of trial-and-error interviewing over 2000 applicants and hiring over 200 for my clients and myself (since ~2006).
Nearly 100% of the applicants were remote, so I think that helped me from falling into traps of poor "traditional" hiring practices.
The point of hiring these remote folks was to help accelerate whatever team I was on. It can be a great way to scale your existing team very quickly if you know how to do it.
For example, I took over an iPhone app dev team and it was taking them 4 months to produce an app. These apps were all very similar in functionality, but the developers were spending a ton of time slicing images, testing, and other tasks that remote workers could easily do. So I hired some remote staff (via oDesk) to do most of that supporting work and we got the app production time down consistently to 1 month. That was a huge ROI for the business since the total cost for all remote staff was the same as for 1 additional on-site engineer.
There's nothing magic about hiring well, but I've watched others try to hire remote staff and the vast majority of them try once, fail, and give up on it.
They will approach the hiring process in a traditional way (personally interview them to watch how they handle puzzles, etc). It's a grueling process and then they still get really poor hires and conclude that "outsourcing doesn't work".
It's most helpful to think of the process as panning for gold. (Naturally, I'm not saying that some people are more valuable than others innately, just that you're looking for those who are most valuable at performing your given tasks.)
So, to find gold, you must filter, filter, filter. That's the exact process for finding applicants that are high performers. Most of your applicants will be pretty terrible at the job you're hiring for, so the filtering process is critical for success.
- filter out the very worst applicants with a small easy question
- filter out the remaining applicants:
- pick a real-life production task you've recently completed
- ensure that the task is *exactly* what they'd be doing in the job
- have them perform the task
- compare their task results to your task results
- hire more than you need of the top performers
- filter out (gently fire) the ones that aren't as good
- repeat as needed until you have gold
When I see others attempt this, the most common problem is that they essentially go down to the river and just grab whatever pebbles they see in their first handful and hope there's gold in it (hire without filtering). Or they go down and carefully pick the prettiest pebbles hoping they will be gold (wrong filter / puzzle interviewing). But the only way to really find gold is to seriously invest in a filtering process that will yield actual gold. That means filtering based on their ability to do the actual tasks they'll be doing on the job.
The great thing about hiring remote folks is that I care 0% how they get the task done. I don't care if they've automated it, or have their mom do it for them, or whatever. If they provide the results I need, I'm happy, period.
There are plenty of other smaller caveats and gotchas to watch out for, but I'll try to cover those in a blog post sometime.
If you're a startup and want to go faster, try this out by off-loading some of the grunt work from your staff. It can be a big competitive advantage if you can do it right.
> - filter out (gently fire) the ones that aren't as good
God I would hate this. I'd rather be told that my work sucks, and shown examples of better work, so that I could actually improve and not just wonder if I was let go for some arbitrary reason. If I disagree with you BFD, life goes on, but I'd at least like to know if it was related to my output or not.
"Gently" just means that you are sensitive to the feelings of the person you aren't going to continue sending work to. That doesn't mean you can't tell them why. I often do tell them why - it really depends on how they took feedback earlier in the contract. If they are defensive and don't take feedback well, then I generally don't give them final feedback since they've made it clear they don't like it. However, if they've taken feedback well during the contract, then I give them final feedback that will help them improve and win future contracts.
The feedback I give is also gentle. Well, I guess everything is done gently. These are all good human beings your dealing with, so why not be gentle? Gentleness doesn't mean you aren't honest with them. It just means that you are considerate of their feelings when working with them.
They may have performed work for you, and they might not have gotten paid. You open yourself to so much liability it's not funny. A judge would look very unkindly to such practice.
OP was clearly talking about hiring people on odesk where different rules apply.
On odesk you hire people for projects of limited time.
He didn't mean that you don't pay the people you've hired if they do the work they promised - that will get you kicked out of odesk really quickly.
What he does mean by "gentle fire" is "don't hire them again for more work".
And that's how it's supposed to work: odesk freelancers have no right to expect being hired by you for second job if they didn't perform to your satisfaction on their first job, for which they were paid the amount they agreed to be paid (just to make this clear).
In practice, it usually means that I start sending more and more work to the better workers until there is little work left for the worker that doesn't perform as well.
I guess you could think of it like a load balancer distributing work. If there is a worker that is less responsive or sending poorer results than the rest, then you start sending less and less work to them. Ultimately, you run out of work for them since it's all going to the higher performers.
At that point, there's no point in continuing the contract with the lower performer. So, then I will let them know and end the contract and give them the best review and feedback I can while still being honest about their performance.
It's worth noting again that these are short-term part-time contracts, so this already fits within the contractor's expectations. They don't expect this job to last forever and they're generally not too heartbroken if a contract ends. They are often working on multiple contacts simultaneously, so mine is just one of several contracts.
Ultimately all contracts end, even for the best workers. The nice thing about hiring temporary contractors is that their expectations are already set that this is a temporary engagement.
"filter out (gently fire) the ones that aren't as good." If the employee finds out you "hire more than you need of the top performers", you end up with a potential court case.
If I found this out about a former employer, I would have pursued them to the end of hell.
Isn't this what the up-or-out policy at major consulting firms and investment banks is all about? I mean, it's not identical, but it's a similar idea. I assume that if sure that if firms like McKinsey are hiring/firing like this, it's not illegal.
Depends on timelines. The big places will keep you on for at least a year, possibly forever. Most small places can't afford to do more than a month on someone they don't like.
Remember that these aren't employees, but are temporary part-time contractors. It's not really "firing", but just ending the contract earlier for some than for others. All the contracts end ultimately, I just keep the best performers much longer than the lesser performers.
IANAL, and I'm just taking a stab in the dark here... But might it have to do with the work contract not being made in good faith?
That is in my basic understanding of contracts it is assumed as part of the contract that there is good will between both parties to fulfill what is set out in the contract. Practices such as these would indicate that the company has no actual intention of hiring the employee and is using the hire as an additional filter.
I suppose if the contract stipulated that there was a trial period or some such it might be a way to wiggle out of it. From a quick google search it seems you can either sue for Fraud (they advertised a permanent position and that wasn't the case, or they didn't make it explicit that the position was temporary) or Breach of Contract if they didn't specify that you were in a trial period.
Would love to hear from someone with better grasp of the situation.
In the US, generally people are employed at will (http://en.wikipedia.org/wiki/At-will_employment) which means an employee can be dismissed by an employer for pretty much any reason and an employee can leave their job for any reason, without any notice.
There's some exceptions (for example, discrimination) but you're free to take a job in bad faith too.
A job being at-will doesn't affect whether you can offer a job in bad faith. "At will" says you have the right to fire me for any reason. "Bad faith" means you shouldn't offer me the job in the first place unless you mean it. They affect two different actions.
Let's make an extreme example: I interview Anderson, Beth, and Charlee. I like Anderson and Beth, but Beth is a better fit for the team. Charlee is terrible.
Meanwhile, All three are interviewing with my competition. So... I give jobs to Anderson and Beth. I offer Anderson a very good salary just to make sure he doesn't go to the competition. They hire Charlee, and that's a win for me. Now I fire Anderson.
I suggest that if this was my plan all along, I wronged Anderson when I offered him a job in bad faith. I may not have wronged him when I used the "at will" provision to fire him, but I wronged him when I fraudulently offered him a job that I had zero intention of letting him earn the right to keep.
In Australia we have strong job protection including unfair dismissal laws which would obviously cover this scenario.
It's part of the balance between keeping a good economy and recognising that jobs aren't just profit generation for employers, but also the means by which people support themselves and their families.
I can't provide any specific laws, but the general objection is that they were hired in bad faith. If I hire 15 people with the intention of firing 5 of them (and don't tell them that), that's a dick move. I misrepresented their opportunity at my company.
It depends what you tell them. If you said, "I'm hiring you" and you expected to fire 5, that's a horrible move. But if you told them, "We have a contract for you now, and might have more work later." Now the fact that you're intending to give 10 of them full time jobs is a very nice move!
The difference is not just semantics either. If they have other work already, the odds that they accept your first contract will depend on what you told them at the start.
This can work if there is a large enough pool of quality applicants that are either currently unemployed or are already contractors. If they are contractors, they may not be seeking full time employment (they like the mobility of contracting). But if you want to recruit someone who has a full time job, lots of luck convincing them to leave a good position (or any steady position) for a 6-month contract that "may" turn into a full time job.
This is how I was hired at my current job though. I was laid off for a few months, and a contract to hire position opened up. I took it, because it was better than not working -- but I would have never considered it if I was working at the time (they ended making me a permanent offer after 3 weeks, instead of 6 months, and I really like the place, so it ended up working out).
I've used a very similar method for myself in hiring remote workers, although my ratios are more extreme. In some cases we tested over 200 people and selected one person for the job.
Have not used math questions at this stage as it seems a bit too obtuse and not to-the-point. Better to ask an actual coding question if you are hiring a coder. However I have had success in hiring people that have been involved in maths olympiads or also in coding competitions.
Let me give you a short answer as a place-holder for a possible edit into a longer answer.
1) "Validity" here means hiring of better rather than worse workers, as all workers can be rank-ordered as to work performance, as compared to doing nothing to select workers.
2) Usually in hiring, it's enough to outrank someone else in a hiring criterion, for example general mental ability, without necessarily reaching any particular score level. The general idea is to rank higher than the other applicants when applying for a job, and to choose higher-ranked rather than lower-ranked applicants when hiring other people.
I'll see if I have time to add specifics later after some activities with my family.
Sorry - I see now that my original questions were pretty vague.
I guess what I'm trying to do is turn your interesting information into an actionable strategy.
Let's try a scenario: Let's say I have a job opening for a senior developer. My goal is to recruit the best possible candidate. I put out the job description, and I get 20 responses. I give them all the same structured interview (.53 validity). I also ask for a work sample (.54 validity). So, in terms of these validity scores, what do the numbers represent? Does a .53 validity say that the applicant who scores better will have a 53% chance of performing better in practice?
Anyway, lets say I somehow qualitatively interpret their interviews and samples, and using some rubric, rank them. (But, how do I know that my qualitative review is even valid?) Then, I give the top candidate an offer.
How do I determine how much better the top candidate is than the 2nd top (to give an appropriate offer)?
Also, how do I know if need to continue interviewing another 20 candidates?
You really shouldn't recommend a "mental acuity" test since those are complete and utter nonsense. I think it's a better screening criteria in the sense that you wouldn't want anyone working for you who believes that IQ tests are worthwhile.
I would be happy to see any research evidence you have found to back up your opinion about mental ability (NOT "acuity") tests. This is an issue that is very heavily researched in multiple countries with varied cultural backgrounds across hundreds of kinds of occupations in almost all kinds of workplaces.
I'm familiar with other books by the author of How would you move Mount Fuji? Thanks for that book suggestion. I have just requested the book from my county library system for use in updating my FAQ document for posting on my personal website.
Where is the line between a coding test and a "work-sample" test? Does a coding test become a work-sample test just by virtue of resembling a task you would do on the job? Is it a function of scale of the task?
>Because of a Supreme Court decision in the United States (the decision does not apply in other countries...)
That's funny that you think this clarification is needed J
(No offence intended)
I failed mathematics in school, for the life of me I can't grasp them beyond the basics, but give me laptop and a copy of Sublime and I'll code anything you want. I can code, but I would fail any mathematical test given to me. This kind of approach has always bothered me, there are a lot of good developers out there bad at maths but posses strong problem-solving and highly analytical skills.
Being a developer is 80% Google and 20% actual coding knowledge. We are hackers at the end of the day, not miniature Einstein's with encyclopaedias for brains.
I've always felt like I was supposed to like maths, but I learned (basic) programming before I learned algebra, and I couldn't figure out how I would be using it.
It looks like Emma's math prowess is working against her. It's ironic the app developers - who need her help the most - are pushing her away.
OK, so there is a difference between computer science and programming. that's why there are two different stack-exchanges:
cs.stackexchange.com
stackoverflow.com
And we can make even finer distinctions if we wanted to.
it's actually really fucking INCREDIBLE that
* you can know tons of CS without being able to build a decent app
* you can a decent facebook clone without having any idea how it works
I feel really bad for Emma. I was a math major, but app developers won't even look at me b/c I'm not a full-stack whatever. So now I'm a Data Scientist at an advertising firm in Puerto Rico.
After reading this I have a dumb question: The person behind the post is a CS major but only played a little bit with the C programming language in college — is this pretty common these days?
Certainly at the school I went to you had quite a lot of choice over what courses you took. I imagine it would be quite possible to get a CS degree without ever touching C, especially if you took either a more math/theory approach or a more UX/HCI approach to your degree. Of course it was equally possible to spend half your degree writing operating systems and programming micro-controllers in C and assembly if that was your cup of tea.
Basically CS is a huge field and any assumption you make about someones skills just because they have a "CS degree" is almost certainly false to a greater or lesser degree.
Yes, I have seen many CS graduates who couldn't write C or C++ if their life depended on it. In fact, I have seen CS graduates who could explain the theory behind, say, parallel computing in detail, but could barely write scripts.
Knowing everything about painting doesn't make you a good painter :).
Yes, I've interviewed plenty of people with degrees from very prestigious Universities that evidenced no real programming skills, and a wonderful (truly) grasp of algorithms. Trouble is, the former is necessary on a day to day basis, and the second is always available from some contractor when you need it.
You've just shocked me! I went to art school and back in the day even if you studied say graphic design or fashion they actually taught you basic foundation skills which included painting and life drawing. Of course that was in the better schools, so maybe CS faces the same issues.
We definitely do C programming at my program (shell, malloc implementation, thread pool, web server, MIPS assembler/interpreter etc.). I go to Virginia Tech, a public university.
I go to an Engineering school, and there are only two classes that use C and are required for my CS major.
The first one is the second programming class we take and is three quarters java and one quarter C. Pretty basic stuff.
The second class is Operating Systems, and is C the whole quarter.
I'm pretty sure those are the only two required classes that use C. And I didn't have to take the first one cause I tested out of it with AP credit, and I didn't do C in HS. So some people will only take one class in C when they graduate.
I started uni in 2002 and only did half a semester in C, and the other half on C++. I picked up a game dev elective for a semester, and did that in C++. Other than that, it was Java and PHP mostly, with smatterings of Lisp and Prolog for our AI classes.
It depends on the quality of the university, I graduated in 1999, but from quick glance at the CS department web site, you still get to use at least C, C++, Prolog, ML, Java, C#, depending on the project assignments.
I believe there is some kind of inferiority complex, we don't believe software engineering is actually worth it. Probably, it is the result of an academic mindset that is taught at colleges, where the applied fields are seen as less important than the "pure" ones. But good software engineering, that is, writing complex systems, with a lot of requirements, maintainable, scalable, nice APIs, etc. it's very, very hard. And we know it!
If we applied our hiring methods to writers, we would be asking them to improvise a rap rhyme, when we wanted to hire a novelist.
My opinion on this is that most of the interview processes is pretty old(over 20-30 years) and back then a good programmer was also a pretty good mathematician, and now most of the people that do interviews just use the same old patterns because, maybe, some of them don't know any better or because that is what they found in some books they have read.
I tend to hate the interviews that ask me to solve math and logic brainteasers because I don't see the value in them regarding my knowledge of programming.
Math puzzles are great if the problem is easily understood, the solution achievable without a math degree, and you ask them to solve it by writing code.
For example: "This database contains 100,000 problems with standardized parameters. The problem definition is defined in the file spec.txt which you can grab from our code repository. Write the code to solve these problems efficiently, passing each solution to a remote service via POSTing to a REST API, the documentation for which you can find here. Bonus points for parallel execution. Feel free to use any editor/IDE and reference online documentation, Stack Overflow, etc. that you want. If anything's not clear or you need a hand with something, just ask as you would if you were an employee already. Ready to get started?"
The great thing is that once you've identified a candidate, you can do remote screen sharing and have them write code before they even have to come into the office. I've interviewed a fair number of remote people this way and it's excellent for weeding out the people who can talk the talk but can't program worth a damn. And it limits bias because you don't care about much beyond their communication ability plus their technical ability.
My observation is that a lot of interviews come down to "stump the chump" questions; a question that is meant to show a single issue the interviewee has under their belt and is used to gauge the entirety of the interviewer's ability. Math puzzles/logic puzzles are in the same category: they require domain knowledge that probably doesn't translate to any job I've ever worked on.
That aside, one must have a way to measure the abilities of a candidate -- and asking the same set of questions to many people allows you to compare the answers as apples to apples.
I generally don't restrict my people from asking any particular question, but I will ask them to consider what a failed answer really means for the specific job (questions are generally adjusted then).
As an aside, some questions of mine that aren't specifically about coding:
* do you code outside of work (a love of coding translates to good coders)
* send me a link to some code you've written that you are proud of (let see what you got)
* tell me about a problem you had where your solution wasn't correct (how have you dealt with failure).
IMHO, the post is more about the interviewers not understanding what is important for success in the job they are interviewing for than about anything else. If you need a person that will have to switch technologies, languages and paradigms, you have to test for that, make sure a candidate has done it before or is capable of doing it in expected time with expected depth.
If one is good and quick in problem solving or has high GMA, that does indicate that he has the capacity to handle new and difficult things in general, but says nothing about the speed with which he can handle a particular new thing. Author's example with JavaScript is very good illustration how difficult can it be to learn a new paradigm for the first/second time.
Anyone who supports math puzzles (or whatever else) in an interview would have to argue that their perception of the candidates performance offers a clear enough data point that it doesn't dilute other information available to them. Given Google's study finding data otherwise, they certainly have the burden of proof.
Dear god, what kind of startup hires a person with only basic Java and Python knowledge, then hands them K&R and expects them to churn out production-quality code?! That's unfair.
Totally agree, asking math puzzles (sometimes really hard ones) to develop a copycat iphone app? Interviewing like this is really off the rails.
I really understand that a startup with scarce resource would like to do its best shot. However as discussed long ago (https://news.ycombinator.com/item?id=2385424), it is really frustrating that asking math puzzles are assumed as the best way to hire the best for the job.
On one hand I completely agree, on the other my experience of most CS graduates is that you can't code, at least not in the way that anyone codes in the real world so it's not a great thing to spend too much time on. That's not the fault of most graduates, it's what and how they're taught. Most people coming out of university know a little bit about a lot of languages and theories. That's good for giving them an overview but not great when it comes to having actual usable skills on day one.
Because of this I've pretty much given up on hiring graduates based on their technical skills so instead I'm looking for someone smart, who gets that they've got a lot to learn, who is interested in technology and can get on with the other people in the team.
I don't think asking people math questions per se is a great idea, but if you've studied a maths degree it's a good way of working out if you're smart and if you were paying any attention at all during university.
(Incidentally this may be different in other countries (I'm in the UK) or in a company where you're able to attract the very best who have picked up really solid skills, but for most organisations that's not the case as most graduates spent more of their own time in the bar than coding.)
If a startup asks you to solve math puzzles, it's possible that the work you will be doing heavily involves the creative use of math or information analysis. (This is more broadly valuable than many people recognize.)
Also, it's also possible that that particular startup doesn't know how to effectively interview.
It doesn't sensationalisticly mean all Startups (capitalization yours) don't know how to effectively interview.
Also, rather than focus on your ability to learn, I would humbly recommend you reconsider the basic nature of employment. An interview should be considered a two-way conversation. You're not selling yourself as a slave, you're entering into a mutually-beneficial, private, voluntary arrangement. Thus, even someone who goes into an interview willing to accept anything and everything they offer could be expected to ask simply, "And what exactly will I be doing?" But better yet, grill them about every nitty-gritty detail you can think of. Although some insecure interviewers may be taken aback (I'm guilty of asserting the interviewer was wrong on more than one occasion, both times still receiving an offer), I for one am impressed when a candidate demonstrates a sharp, critical and skeptical mind in this way.
Even after reading Jeff Atwood's post[1], it still amazes me how many programmers fail the fizz buzz test. We dont even get the chance to ask tough programming qustions. Simple questions like fizz buzz, loops and recursion were good enough to filter out a lot of applicants.
Is this really the case? I read this a lot, but have a hard time imagining that there exists a wealth of people who have such a hard time with such a simple task, and yet I am having a hard time finding an entry-level job when they are (presumably a substantial part of) my competition.
It is. There are a lot of really bad programmers out there.
However, you're probably not looking at the places where such programmers would apply... try an entry-level job here in Uruguay (pay: about U$ 800 / month after taxes), you'll get a lot of people that fail fizz-buzz .
The original poster (Imran Ghory) was from the UK, and Reginald is from Canada. Neither are Silicon Valley.
This may be another reason people are eager to start their own company in lieu of working for someone else. If the questions are rubbish and completely unrelated to the actual job, then there's a huge disconnect between the interviewer (or HR company, as a lot of places outsource that) and where the actual work is to take place. I blame both.
The irony is that, in an effort to hire the "smartest" people, they leave out the wisest. Which is arguably more useful.
Well, if they really enjoy the challenge (and it is challenging), then yes. But I think there's something sad about being forced into it because hiring practices are archaic elsewhere. It doesn't have to be this way.
I don't like it that so many new startups fail and I have a feeling many of these (besides lacking an idea, failure in implementation etc...) are due to a lack of other options. If you can't get hired with the skillset you have, even though it should be more than enough, it can drive you to do desperate things. Including becoming a founder.
I'm sure there are a certain amount of people who have simply given up on or refuse to subject themselves to the standard interview process and will either become freelancers or get their jobs through networking.
Depends. They will likely work on the sort of problems they are good at solving. But at the expense of spending a large part of their time running a business, rather than solving the problems they are good at.
1. Most IQ tests are Bullshit
2. We all know what happened to the company famous for " Who moved mount fuji"
3. Math Puzzles are good if they are of the IMO level- but these things need a lot of concentration and joy to solve- Not under stress interview conditions.
4. Expecting someone to show brilliance by solving a math puzzle in under ten twenty minutes is a lot like a public willy wagging competition
5.Even more disgusting is the semi dumb questions at Mckinsey inerviews like - "Estimate the number of mineral water bottles in London"
6.
7.In 'Jobs', Walter Issacson says Steve was never into much of these puzzles- I can understand the reason.
8. ' I think a lot of what people call intelligence just boils down to curiosity'-(great quote from an inspirational friend-http://www.flickr.com/photos/elizabethbw/8373942339/)
9. People who ask these kind of puzzles end up creating a lot of CPU without any GPU. Very Little beauty. Very Little love.
Disclosure- Im a member of Mensa Inernational. No Offense meant.
- After a first non-technical call, we ask the candidate to create a very small project based on our SDK. We send him the documentation and a very small sample. He can almost use every tools he wants to create that small project and, of course, we do not set any deadlines. It allows us to see how the candidate architecture his applications and it gives us a project to discuss during the following call.
- If all goes well, we invite the candidate on site to present our code/project and eventually brainstorm together. So that both parties can see if they can work together and the candidate has an insight about how we work, how our code looks like.
Clearly, it's far from perfect and we are often considering changing it. Imagine if every company where you are applying would ask you to create an app from scratch with their SDK? We may lose some candidates, but at least we hire only people that fit the company's culture.
I have experienced "interviews" like this from the other end. I told YC funded founders I'm not interested in their job if this is how they interview. In my opinion, it shows inability to make a decision.
I am an excellent developer, with years of experience in the industry. I know lots of technologies, and already have a great job. There is no reason for me to spend personal time writing your projects, when I would be rewarded by spending personal time on my employer's projects.
"Interviews" like this will only grab candidates with nothing better to do than to fulltime interview with your company. In my opinion, the best people already have jobs, and you're excluding them from the process.
"I am an excellent developer, with years of experience", but seemingly unable or unwilling to complete a small practical test. Sounds to me like they are good at weeding out the wrong people.
What makes you think "I work at Google" provides sufficient data to an interviewer such that they should exempt you from their interviewing process? The fact that you have a job somewhere else says very little.
I'm not saying I'm immune from interviews. I'm saying it's proof I have chops.
Technical interviewing in a broken process. I've given almost 200 technical interviews at Google, and I've seen all kinds of results. But, I believe Google's results. Having worked at Google longer than anywhere else I've ever worked, I can say that the people are incredible, and it's a direct result of the interviewing process. We interview someone for a set number of interviews (N≤8 nowadays) and we make a decision. I can count on one hand the number of people I think are deadwood.
All I'm saying is the "interview" process of having people do projects for you is broken. You will filter out a lot of people with jobs they are kicking butt at.
This is sadly common in a community keen on logic, evidence, and avoiding fallacies in thinking. Worse than puzzles are pointless faux psychological screening questions like "Tell me about something painful that has happened to you and how you dealt with it."
I would (and have) asked if the interviewer or organization has any evidence to show that interview puzzle performance (or shit like Myers-Brigg) predicts job performance. No? Not surprising. Google did look into it and found no relationship. (http://www.businessinsider.com/how-google-hires-2013-6)
Programmer interviews are so crazy and sometimes sadistic that I catalogued some of the more common interview patterns:
While I agree that puzzles and mind games are silly ways to appraise coding skills, they do give an insight about a person's raw intelligence, or knowledge, or potential. As CS is an application of math and programming is an application of CS, being good in math does not necessarily mean proficiency in it's application; same goes for CS. IMO, a good programmer must, at least, have:
+ knowledge - generally mastery of math/CS concepts and can be thought of as the potential
+ application skills - modeling a real world problem into a theoretical, computable, and (ultimately) programmable form
+ execution skills - implementation (coding) of a solution including the ability to utilize requisite tools/technologies such programming languages, DBs, OS, and so on
That said, hiring process should cover each of these areas and programmers should work on all these as well.
I have the exact same problem as OP. Getting tons of job offers because I've been doing competitive math and algorithms since grade school but really have hard time understand technology. I'm pretty ambitious and I want to join a small, high growth startup and have the excitement of being part of a founding team but I'm afraid of letting people down.
I could learn heroku/RoR/whatever other technology but news things are always coming out and some people keep up with it so easily. I'm not sure being a dev is right for me if I take so long to understand such basic stuff. But I love coding and algos! I write python scripts to do all my homework... and then run them in codecademy labs because doing it in unix makes me so confused.
If anyone has had the same problem please let me know how you got over this hurdle. Thanks.
You say that unix is confusing. Unix isn't confusing. It's just unfamiliar to you as it was to me at one point. The way you get better is that you push through the unfamiliarity and keep going until you solve the problem you're facing. Sometimes it's painfully slow to acquire the knowledge you need, but that's just part of the process.
All that knowledge accumulates over time and you build skills about how to learn new things.
That's what allows people to pick up new technologies easily. Hard work, previous experience with similar technologies, and some intelligence (which you obviously have already).
I would recommend that you install Ubuntu in a virtual machine (VirtualBox is a good choice) and see if you can get some Python programs running. That process alone will teach you a lot.
For what it's worth, RoR and the Heroku stack are not basic and you shouldn't feel bad for not picking the up quickly. There's so much context that you need to have to understand what's going on and to use those technologies effectively. I bet if you polled your classmates the majority of them haven't even heard of Heroku.
When I hire programmers I try to favour ingenuity, knowledge and as best as I can gauge it, work ethic. For instance I might ask them about a theoretical task, possibly something like a scheduler or packet filter etc and give them domain specific data about how it will be used and ask them if there are any optimization they could make if they had this data about the systems use case.
Or I might ask them to describe how an event loop works.. or what the I/O path between their program and the disk looks like in as much detail as they can.
Someone that loves the field is going to have a decent idea about these things even if they never had to build one before.
caveat: these examples are very system level but you can substitute them with appropriate web, financial etc domain specific knowledge.
Finally some one pointed out the importance of the ability to actually code and produce something that works. Algorithmic problem solving ability is far less utilized in actual every day job compared to being able to code. Just imagine how much of your math skills did you actually need going well through all those bad experiences? Would you still be considered slow learner and fired if you knew how to code pretty well but just wasn’t so good at figuring out shortest path in a graph. Isn’t it possible to know the CS basics well i.e. familiar with complexity, big Os, basic data structures and sorting and being able to learn any advanced standard algo when needed by looking it up? Just wondering….
I think a lot of companies use brain teasers or math problems to test general mental aptitude, whether it works or not is under serious debate, but in my opinion that's the right thing to test for (if it's even possible).
The reason is smart people can figure out git, or databases, or objective-c, or whatever, in a fairly short amount of time.
For example, my co-founder learned objective C off free online video tutorials and built an iOS app (talking an app with serious firepower and back-end transaction logic) from start to end by himself in less than 3 weeks.
That's why we're not as concerned about what you know right now as what it's possible for you to learn in 3 more week.
I can relate on the opposite. I am not great at those complex math problems. But, I have been coding for 15 years at >20 hours a week average. Mostly web stuff. I've built dozens of full products, that we're complex, and I generally feel like I could build anything I wanted. Every time I use a new site I can visualize how I would have built it, usually not a question of if I could; time permitting.
Yet, I have never had the balls to pursue it professionally. I build stuff and usually never launch it. I have learned several times over that marketing is not my strong suit.
That said, I'd actually like to work for a startup. Hit me up if anyone wants to talk.
If you're running a startup, the most important thing to hire for is fit. Do they fit in your culture? Do they fit a need that will help you achieve your milestones? Do they fit in the overall growth trajectory of your company? Do they have competency in the specific area you are hiring for and/or where your startup is building overall competency? Can they manage themselves and their time well?
The likelihood of failure of a startup approaches 100%, so you should optimize for likelihood of survival, not for IQ.
If you're not a startup, then the top ranked comment applies. But it doesn't really otherwise.
Except that what a lot of startups mean by "fit" is "they think just like me" or even, "they have a similar background to the rest of the team." Sure, that may make them easier for you to talk to or hang out with, but it reduces the diversity of the team, and increases the likelihood of groupthink. The other things you mention (after "fit in your culture") are not what most people mean by "fit". "Fit" is an intangible that can be used by screeners to keep from going out of their comfort zone.
The interview is a 2 way street. While the company makes you jump through hoops to see if your good enough, you also have a opportunity to determine if you want to work for them.
I like to ask "what will I be working on in the next 6 months" that way you don't rock up and than the second day they through you in the deep end of building a iPhone app.
Granted, startups only have a vague idea of what they will be programming with short periods but it helps.
Also ask "what will be my performance indicators". If they don't include "being able to very quickly learn new technologies" its hardly your fault.
In my experience, only the really big companies focus heavily on algorithms and math puzzles. That's because they don't really need to hire anymore people, they just want to steal 'smart' people, and they don't need to iterate as quicky. Start-ups and smaller companies have in my experience, typically asked full-stack type of questions that dive into things like networking protocols, databases, scalability, and so on. And I believe thats the way it should be. Start-ups that focus heavily on math puzzles and algorithms are doing it wrong.
The "Kevin Bacon" stuff was about degrees of separation (Does the expression "Six degrees of separation" ring a bell ?).
Not long ago, Facebook made that 4.74 degrees of separation on its networks. Meaning a maximum of only 4.74 persons are necessary to connect any two random persons on the network.
I found that Facebook was really bad for this. I'm a math undergrad, and I applied for a bunch of data analysis positions with them.
I was asked, as part of my application, to take a programming quiz. The quiz consisted of a graph theory problem. I did pretty poorly on it, given that I have no real knowledge of graph theory.
Had they asked me a question about statistics (or something similarly related to data analysis), I think I would have actually been able to answer, or at least been at a point where my programming knowledge- not my math knowledge- was what was holding me back.
I would say that you should keep at it. There are strong parallels between math and programming, but interviewers should definitely be asking you to write pseudo-code on a whiteboard and do a paired programming session. That would probably be a good way to relieve the awkwardness later when they realize that your programming skills aren't as strong as you'd like them to be. Definitely keep at it, soon you'll be able to think of a Markov chain as a for loop multiplying two arrays and not only as a matrix multiplication.
It is a good strategy, if the company is interviewing freshers as programming is teachable and the assumption is that new inductee will take few months to become productive. If you can't wait, the best strategy is to give a live coding problem and test the person's proficiency in the required language/technology. I invariably do the latter as my requirements are always very specific. Most start ups I suppose, are themselves undecided on product/market/technology choice and thus the former strategy.
This is a great post. And also from the other side of the fence. As typically we see, these kind of posts, from people who did not like Math puzzles, and as a result suffered in the interview rounds.
But this one talks about getting inadvertent benefit of being good in Maths to get selected for programming, and suffering the consequences later on.
Also, it highlights the importance of what is mostly taken for granted and thought of as mundane stuff, of programming - the idiosyncrasies, jargon, and best practices of various languages and OS environs.
* CS majors take math in college, but there's lots of other stuff going on with their CS classes so it's hard to remember math class. If you can actually remember anything from math class, you must be smart!
* Programmers like math, even though it's not their job, so interviewing is the only opportunity to have fun with it!
* We had to deal with this crap when we were interviewing so now it's time to make someone else suffer! >:)
While we're at it, can we stop treating and thinking of web development (which seems to be a lot of dev positions these days) like it's rocket science?
I am willing to bet that tests give an advantage to applicants with no commercial experience, as well as those that have no life outside of technology. If that's what you want in employees then sure, it's a good way to find them, otherwise just look at their code samples and give them a trial.
Programming isn't difficult and you don't need to know complex maths or be able to solve mind bending puzzles to be a great developer.
I am stopping to talk with people who ask for such a bs.
Check out the last technical interview task that I got
```
Objective:
Write a program that prints out a multiplication table of the first 10 prime numbers.
The program must run from the command line and print to screen one table.
Notes:
- DO NOT use a library method for Prime (write your own)
- Use Tests. TDD/BDD
- IMPRESS US.
```
I mean I can impress you but how will this correlate with production code?
Something I've always wanted to ask, are contractors hired the same way? I've never contracted although my father did in his retirement years. I'm curious if modern contractors have to put up with this kind of behavior at interviews, or if its a more professional atmosphere oriented around the actual job requirements.
I'm absolutely horrible at math... I think I graduated high school (my only schooling) with the equivalent of just above grade school in math... I can code no problem though. I'm very good at it. Anybody asks me math puzzles I say thanks for your time, but I am done with the interview.
I don't get it. He got hired? He learned how to do his stuff? If we require people to know how to work right out of college, no fresh grad would ever got a job.
Sadly, it's not. I can't tell you the number of 5-8 years' experienced candidates from respectable companies who couldn't solve very easy interview problems.
Yes, I get that a whiteboard isn't an editor, and you don't have google/SO, and some people just freeze in an interview, but I've long ago stopped assuming people can code just because they've got coding on their resume for successful companies. (For all I know at the interview moment, they didn't have a coding role, or may not have even worked there.)
Stop asking this fine young lady math puzzles to determine her programming abilities. She is good at solving your seemingly pointless math puzzle, because she was practicing problem-solving since she was ten. But she is not anywhere near as good at programming, yet - which caused her problems at the actual jobs she had to do after she was hired.
Because most paid coding work is not public, and many excellent commercial programmers prefer to do other things in their off time.
Honestly, this is a weirdly naive question. Gung-ho startup culture does not represent the industry as a whole, and you don't need to eat, breathe, and sweat code every waking moment of your life to be a great coder.
So it boils down to "show me your code" and then "please write a few test examples". To staff up a cheap coding sweatshop this method is good-enough.
In most cases an applicant must be able to read English (to google some code to copy-paste and occasionally search through documentation) and able to install and run Eclipse.
The real problem with hiring is that a HR middleman is ignorant and can't tell a good code form a restaurant menu. So he must give a very few simple exercises from common text-books with known answers.
The even bigger problem is that almost no one needs coders, everyone wants programmers which is a complete different set of analytical and engineering skills.
Coding is just a process of translation of a ready-made by someone else, poorly understood (if at all) specifications into a spaghetti [Java] code by calling poorly understood methods of ready-made classes, coded by someone else.
Programming is a process of understanding and describing reality (in terms of design documents, protocol specifications, and then, least importantly, source code in a several languages).
The criteria of success for a coder, btw, is when it just compiles (unit-tests? what unit-tests?) by the industry-strength most advanced compiler of the most sophisticated industry standard static-typing language (static typing is a guarantee from stupid errors, everyone knows) which is even verified to run correctly on the most advanced VM which incorporates millions of man-hours of optimizations, unless.. Never mind.
Success of a programmer is when it, like nginx or Plan9 or OpenBSD, is good-enough.)
The author was not saying she was above simple coding tests but that she was an inexperienced programmer who was GOOD at the maths puzzles and got the job then struggled and the interviewer's assessment of her software development ability would have been better with some software related questions.
Before the interview, I ask them to write some code to access an HTTP endpoint that contains exchange rate data (USD, EUR, GBP, JPY etc.) in XML and to parse and load said data into a relational database. Then to build a very simple HTML form based front-end that lets you input a currency and convert it into another currency.
I ask them to send me either a link to a repository (Git, SVN etc.) or a zipball/tarball. If the job specifies a particular language, then I obviously expect it to be in that language. If not, so long as it isn't in something crazy like Brainfuck, they have free range.
If the code works and is basically sane, that goes a long way to get them shortlisted.
During the interview, I'll pull the code they sent up on a projector and ask them to self-review it. If they can figure out things that need improving in their code, that weighs heavily in their favour. Usually this is things like comments/documentation, tests, improving the structure or reusability. If it's really good, I'll throw a hypothetical idea for refactoring at them and see how they think.
The reason this works is that, despite Hacker News/Paul Graham dogma to the contrary, "smartness" isn't the only thing that matters in programmers. It's actually fairly low down the list. When hiring programmers, I want people who are actually able to do the daily practical job of writing code, modest and self-critical enough to spot their own mistakes, and socially capable to actually communicate their decisions and mistakes to the people they work with.
I interviewed a guy who was intellectually very smart and understood a lot about CS theory. I asked him why the PHP code he sent me didn't have any comments. "I don't believe in comments because they slow the PHP interpreter down." Sorry, he can be smarter than Einstein but I ain't letting him near production code.