- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.

- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.

- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)

- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.

- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.

- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)

Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?

A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.

If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.

But compared to coding and math, puzzles feel sterile. I don't get to build anything, and I rarely get the kind of insights I would get from figuring out a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even cryptanalysis is more fun, because there's a real opponent.

I know lots of amazing programmers who love puzzles, and that's cool. But if you only hired puzzle lovers, you'd miss out on a lot of good people.

Everyone likes to use Google as the example of the elite programming company. I've read the books about them, used the products, followed their progress over the years. So, sure, a part of their big success is hiring the 'smartest' college grads.

However, I've also worked with Ph.D.'s and even interviewed them for positions lower than ones I've held (I have not yet finished college - some day...). And I can tell you that book smarts != execution smarts.

I feel the same way about puzzles. I have never used puzzles or asked 'small' one off programming questions. Instead I've always been interested more in grep'ing the concepts and if the personality is the proper fit. In my mind, anyone can open up a book and read about specifics about O(log n). But how does a person persist at solving a problem and how do they (can they?) change direction if their solution sucks?

I feel like I value creativity over anything and they can creatively solve a puzzle, cool. But if they can creatively solve a task on the system I'm building and are a great fit to the team, then better.

I'm an API design geek. I've never seen a cool puzzle about API design, or something remotely related.

also, replying more to the other post above - you can get insights from puzzles. a lot depends on the puzzle (which is why ita is getting praise here). for example, the ita puzzle i remember taught me a lot about handling data in large numbers of dimensions (in particular, it made me think about the "curse of dimensionality" which was something i was aware of before, but hadn't considered in detail; it also led me to understanding some tools for dealing with it - locality sensitive hashing being the most interesting at the time).

Lattice animals include polyominoes, which are probably familiar to most HN readers, but subsume other graph-related objects as well. I didn't know much about these, and ended up reading a bunch of cool papers on them as part of working on the puzzle. So at least for me, good puzzles have catalyzed some learning.

My mathematical preference is continuous math; my preference for using mathematics is to apply them into real problems.

I'm not saying that all puzzles are pointless, but I think most interview puzzles are kind of an empty exercise that only a certain type of person likes, and that only people who know the trick will get.

Of course, we couldn't easily do A/B experiments, so we'll never really know how the true attract:alienate ratio.

Trying to look at it objectively, I would probably be somewhat put off by being required to do a puzzle to "prove myself", but would then find the specific puzzles offered to be pretty interesting, and worth playing around with. Perhaps it's not tantamount to a $300 tax on ones life it is seemed fun and valuable in its own right?

Note: edited typos.

It is much more like having to travel 90 minutes each way to get to where they want you to interview. That could be very offputting if you do enough interviews that three hours each adds up to a big number, but at least you don't need to worry about them being tempted to rip you off.

During the same job search, I also noticed Facebook had puzzles. I solved one and got invited to interview but never went. I wouldn't have applied otherwise. I guess this is another lesson - if you put public puzzles out there, expect people to send solutions without necessarily being excited to work at your company.

I'd never seen it before the first interview. In that first interview, I broke it down quickly into its component parts, and figured out where the trick had to be -- and from there quickly solved it on the spot.

In the second interview I spent a couple minutes trying to remember the solution from the first interview, and completely flubbed it. It wasn't the first part of that second interview that I flubbed, either, including arguing with one of the interviewers and basically telling him he was wrong.

Oddly enough, I got the job with the second company and worked there for almost six years. For the record, I love doing those kinds of puzzles. But I do video game development, so maybe puzzles are a bit more relevant to my industry (in that 80%+ of what we're doing is figuring out how to solve puzzles in the form of how to do something with less CPU or why a particular video card is behaving in a certain way).

I don't think you give yourself enough credit in the success of the puzzles. They were really well-conceived problems - just difficult enough without requiring too much domain knowledge. Very 'catchy', if that makes sense.

If I were asked such a question in an interview, my response would be: "Dunno. Let's talk about some real-world problems you're facing right now instead."

I dislike trivial questions, especially during an interview. I'm a professional, not a circus animal.

Bingo.

To me, this is the kicker statement for the industry. Solving fake problems like nickel-in-a-blender is for the clever. But I sure as hell don't want clever code solutions; I want rock-solid, reliable, fundamental solutions.

Anyone who feels these questions are an indicator of programmatic success is measuring the wrong things.

I sit on an engineering hiring committee and I can't really think of the last time I saw one of the ridiculous puzzle questions show up in an engineering interview. The WSJ article was pretty ridiculous.

Sorry, but I'm not familiar with the WSJ article you've referenced. Link?

i have done your puzzles for fun, but i would still be annoyed at having to do half-baked puzzles if i were suddenly out of a job and applying to a bunch of places.

(ps. and thanks for them :o)

We also never put a puzzle out to the world until a bunch of us had solved it. There was a long gestation period, and there's a "puzzles-discuss" mailing list for working out the kinks (and, in many cases, rejecting certain puzzles entirely). So, yes, you have to make a real commitment to do it well.

On the other hand, developing the puzzles was a lot easier than most of the coding on our real system, so it's all relative. It was actually really fun to do this stuff, and a nice change of pace. One of ITA's best developers (Justin Boyan) would take it upon himself to create the "perl postcard" solution for each problem. Another (Jim Rees) would always make an incredibly fast C++ version; it usually ended up being not only the fastest of ours, but faster than all the submitted solutions as well. I learned things just by reading his puzzle solutions. :)

I found the process rewarding beyond the hiring benefits, and I'm sure I'm not the only one. (Well, I hope so, at least.)

In any case, kudos and thanks to you - those puzzles were a lot of fun, and I always wondered where they came from.

So, here is my opinion as a non-halfway-decent programmer.

I thought puzzle was fairly fun, but I like programming for two reasons:

* Building useful knowledge. Learning a new language, library, or algorithm I want to play with, programming is the natural way to learn.

* Building cool stuff that people like. Having an awesome product when I'm done with it gives the feeling that I've done a good job and produced something of value.

Programming puzzles don't really do either of these things for me. My motivation for spending three hours doing a puzzle is that I would like to talk to a person. I am basically assured that the puzzle will be used as a filter for the company to avoid talking to people. It will run against a test suite and if it fails it will not be looked at by anybody. This seems very impersonal. Is this the experience I should expect from the rest of the company if I'm hired?

I've got a Git-hub account with some of my code in it. I've spent time on it and enjoyed writing all of that code. I published it to the internet so I'm (at very least) not ashamed of it. Why can't you look at that to determine if I can program? Why do I have to write code specifically for your particular company?

This isn't an attack on ITA, it seems to work for them. But they are a unique case of a company where it seemed worth it for me to spend the time to do the puzzle. Any other company I would have just skipped it unless I genuinely had nothing better or more interesting to do (of which I almost always do).

When I applied at ITA they didn't even acknowledge the program I had written to go along with the application. Not even a quick 'it ran' or 'had the right output but we hired somebody else'.

I'm glad I didn't get the job because not having enough time to evaluate or even run a programming test you gave is also a pretty good counter-indicator of a good company to work for. And when I wrote back to express this the guy sheepishly admitted they had so many exceptional applicants so they just went with an Ivy-league graduate. Wow.

If you are going to require a programming test, be prepared to evaluate it.

I agree. It is possible the reviewer didn't think your code would run, couldn't get it to compile, etc. But, nevertheless, you should have received a fuller response than that. If I were still COO at ITA, I would ask someone to look into your particular case to understand what happened.

I gather that someone with a stronger theoretical CS background than myself would have spent an hour looking at the puzzle, said “Ah, this can be modelled as a graph and solved by applying the Flibblewhiz Heuristic!”, spent another hour coding up a first draft of the solution, and a third hour optimizing and cleaning up the code.

My ego has survived the blow of learning that I am not awesome enough to work for ITA, but I wish I had figured that out sooner.

I was thinking more of all the cool search problems like Strawberry Fields, Traversing the T or Palindromic Pangram, where coming up with some solution is easy, a naive exhaustive search for the optimal solution was not feasible, and where verifying the validity of any heuristics / pruning was not necessarily trivial.

I worked on that one for a looong time, and what bugged me was the screenshot of the expected interface had a result returned in like 0.05675ms or something, and I couldn't figure out how to get it that fast. It took be forever to convince myself that it was just a mock and probably should have read 0.05675s (seconds instead of milliseconds).

We often had internal solutions (and externally-submitted solutions as well) that were orders of magnitude faster than the average. There was a very wide range of execution times. E.g., we had solutions to the Queens & Knights puzzle (in the archive) that ran in tens of milliseconds, if I recall correctly. Typical solutions we received took orders of magnitude longer.

Usually when one solution was a million times faster than another, it was because the fast solution exploited a clever algorithm or data representation -- e.g., representing an 8x8 chess board mask as a single 64-bit register rather than as an array of bytes.

(The sub-millisecond time was for real)

I may have missed that the time was the time taken for the search algorithm to find results, I think I was under the impression that should be the time for the request/response to happen. Hmm.

Many of the articles about Google interview questions on sites like Business Insider, at least for software engineering positions, are total bullshit. http://www.technologywoman.com/2010/05/17/debunking-the-goog...

But the programming puzzles on the website were just awesome, even though I just solved them because they were fun, not for applying for a job. I've been surprised that nobody else has done the same thing, but your explanation of how much work went into the puzzles kind of explains that :-)

I did one of them just for fun 3-4 years ago: "implement a scalable multi-user chat server without using a framework like Netty/ACE/EventMachine". Unfortunately, at the time I couldn't move to Boston, so I never submitted my solution.

Likewise there were puzzles that would appeal to web developers (e.g., build a typeahead web applications), machine learning/AI folks/etc...

But the puzzle process, as originally conceived, (in 2000!) was not necessarily supposed to be a hard filter. As an applicant, you had several ways to stand out, and doing a great job on a puzzle was one. Having someone who already worked for us say you were awesome was another. Being an open source superhero was another. Etc.

In actual implementation, though, it probably was/is used as a hard filter.

It always becomes a hard filter, there is a lot of ego in development and test are an area where ego's shine through. It's my major beef with quiz exercises in interviews, they always become the hardest filter, no matter how much we try to contain them.

That's a very good point that I've not seen made so well before.

After so many failed Google interviews, I found it easier to build a company and sell it to them, instead.

I would argue that they don't and I believe that it is due to their hiring process. Now I do believe that they have very talented individuals but there organization reflects their interviewing bias as such so does their product portfolio. There is a reason that they have to acquire new products for their portfolio and that reason is that they are not recruiting the kind of people that generate new products. As such their hiring process should be suspect.

37signals hires for business value because, as a small company, it has to. Google is a huge company and doesn't have the same incentives. Google's size can blind it to the fact that its hiring practices optimize for mathematical knowledge and intuition over business value. And, as a business, Google is wrong and 37signals is right.

I don't believe that's the case. What I've seen with talent acquisitions (at Google and elsewhere) is that interviewing the team is actually prerequisite to making the acquisition. Of course, the interview doesn't have to be as rigorous: during the due-diligence process you can actually view the codebase and see the individuals' contributions.

What I've also seen, specifically in the case of Google, is that individuals from acquisitions would be asked to stay a year, at the end of which they'd need to interview for other positions at Google: if they couldn't pass the interview, they'd be let go with a very generous severance package.

That said, I am sure there are exceptions.

(Also, I've never heard of G+ being in any way a product of a talent acquisition.)

Generally whenever I've done this to recruiters their interest level goes up and they tend to be far more willing to listen to your concerns about the interview and go to bat for you. And recruiters tend to have a lot more ability than you might think in getting parts of the technical interviews largely ignored by hiring managers - including at Google.

Incidentally my impression from having talked to several Google recruiters is that a decent number of them are incredibly frustrated with the way the process work, and will work hard to get you past this hurdle if you want them to and provide them with enough ammunition (e.g. explain why the question was inappropriate for the position).

Nope! Instead they gave me an hourlong verbal interview, which went great, took me to lunch with the team, which went great, then sat me down with a lead developer, who had printed out some of the winning entries from the obfuscated c contest, and asked me to read them and tell him what each program would do when compiled. That did not go so great. I didn't get the job.

That experience remains burned into my brain, and as a manager I won't give puzzles to an interviewee. I'll ask to see code and expect them to speak intelligently about it & the choices they made, but I won't ask them to write code (or, compile code in their head) on the spot.

(Postscript to my story -- that company never shipped a Mac email client. I believe they interview-filtered themselves out of the entire Mac market.)

Mainly because I do bring in lots of source code. I write tonnes of code in my free time. I contribute to open source projects on occasion and have released some too. It's all out in the open. All my blog posts, everything. I'm an open book when it comes to my ability, what I've worked on, etc.

Yet 95% of the interviews I've walked into the people sitting across the table from me had to look at my resume while they were shaking my hand to find out my name.

Then the puzzles come in the hopes that I will be filtered out quickly so the person interviewing me can get on with their day.

I also find that in such an intense situation my brain is hard-wired to find the "right" answer so that I get what I want without getting burned. It's kind of hard to have a jovial discussion about code and logic when you've got a lot riding on getting the job. It's almost Pavlovian.

There's always the probation period to see if they can actually write code and solve real world puzzles. I've never met the mythical applicant that could bullshit their way through a technical interview.

It took me a long time to appreciate the utility of something like FizzBuzz in terms of how Pareto-optimal it is.

I think it finally clicked when I was getting interviewed for my US citizenship by USCIS in 2008. In the preliminary stages of the application process, the officials had gotten us all psyched up about how there is going to be a US civics test and an "English test". We were given a 100-question booklet of Civics 101 and a lot of anticipatory anxiety.

So, imagine my surprise when the interview came around and with it, the storied, fabled English proficiency test. You know what it was? Listening to one simple sentence read aloud and transcribing it accurately. It wasn't even a hard sentence, it was something on the order of "The man walked his dog across the yard and over the turnpike". Really? Really?! I was flabbergasted. Come on, I could pass this in like 4 languages. One sentence? That's all?

After thinking about it, I realised there was more utility to it than I had appreciated, though. Yeah, I've been speaking English for 18 years, but there were people in the room who literally did not. They could not pass this test. And for someone who really cannot speak, write or understand English much, this test would be genuinely challenging or impossible to pass. At the same time, 90% of people who can pass it probably are sufficiently proficient in English to function in this society. I'm not arguing that it's a particularly insightful, perspicacious or revelatory test, but the point is that it accomplishes more than you think, from a statistical perspective, even though anyone who is even marginally literate would dismiss it as ridiculously easy to the point of absurdity.

So it goes with "a priori" programming tests, I think.

But every time I read "FizzBuzz", I drop everything and go back to refactor my latest version, aspiring to get it down to one conditional.

    >> (1..100).each { |n| puts `curl -s fizzy.heroku.com/#{n}` }

I revisited my FizzBuzz just now and threw together this obscene Python one-liner.

(If I write this kind of code in production, I should be a guaranteed "no-hire"!)

    from itertools import chain, combinations
    fizzbuzz = lambda n,terms: ((lambda d: d.update((x%d,w) for d,w in sorted(reduce(lambda (d0,w0),(d1,w1):(d0*d1,w0+w1),sorted(x)) for x in chain(*(combinations(terms.iteritems(),s) for s in xrange(1,len(terms)+1))))) or d)({0:str(x)})[0] for x in xrange(1,n))

    print ', '.join(fizzbuzz(100,{3:'fizz',5:'buzz'}))

After all this FizzBuzzing I guess I should start writing one up just so I can say I've done it.

But when I'm interviewing, I do ask people for a pointer to their open source projects. And if they don't have open source projects, I give them a workstation and ask them to write code. I've seen too many self-proclaimed programmers who claim, "I spent the last 2 years writing Python," who can't sum a list of 10 numbers without using Google. (Hiring can break my heart.)

One way or another, you've got to see code. Real code is best, but it's better to ask people about toy data structures than it is to hire 3 team members who can't write FizzBuzz.

I did a hiring round once for my old team when I worked at Yahoo years ago, where out of 10 or so CV's the recruiters passed me, about 5 people who were given a short list of screening questions by e-mail for them to answer at their own leisure produced answers that were obviously plagiarized from various online sources.

I got suspicious when one guy presented me with two pages of nicely formatted answer complete with section headings that didn't make any sense to a question that should have required a one line answer. He'd copied and pasted the whole thing from an on-line copy of an Oracle manual, so I started doing searches of random sentences from their answers.

Another one managed to cut and paste an answer that was flat out wrong from a forum post answering pretty much the same question, and hadn't bother to read further in the comment thread, where several people corrected the answer he'd used...

It's when they can't even answer simple stuff WITH using Google it gets really depressing, and it happened regularly enough that using screening questions by e-mail was a real time saver in weeding out undesirables, despite the ease with which people could "cheat".

My favourite STL-related question to ask is simple and can be done on a whiteboard in minutes: if you didn't have std::set<class T>, but you had everything else in the STL, how might you implement a set<class T> class?

There's an ideal answer[1], and then there's people who start thinking about low-level implementations. The ones who I don't want are the ones who don't have any answer at all. I'll continue to be interested in anyone who starts talking about the way to solve the problem. What matters to me is not any code that you write (although that's interesting) but your thought process. After the candidate has worked through this, if they haven't gotten the ideal answer I always give it to them: this isn't some national secret, and seeing them process the ideal answer and realize that yes, it really would work is a pleasure.

Sadly, there are cases where it doesn't work. I had one candidate who did very well on the phone screen, but didn't know the STL. I looked up what he claimed to know (Qt) and reframed the question in terms of Qt's classes—and he still couldn't come up with any answer whatsoever.

-a

[1] The ideal answer is precisely such because you're writing much less code and using stuff that people smarter than you have written. It's to use a std::map<class T, bool>.

Not everyone has performance anxiety, but I've met plenty of programmers who do. Unless you spend the first 15 minutes of your interview calming them down and warming them up with simple technical questions that put them in the mode of programming, you might as well be asking them to sing the star spangled banner naked in front of an audience.

Trying to evaluate programmatic skills is surely a challenge, but I've found a method that seems to work very well -- simple walkthroughs. I'll do at least two walkthroughs of code, asking a candidate to explain to me what's happening using two different artifacts:

* Some code they've written that solves some problem

* Some code we've written that solved some problem

This has proven to be quite effective in rooting out those who look great on paper and talk a great game, but fall apart when put into action. On the flip side, I've yet to find someone who could explain their code in sufficient detail that then proved incapable once in the job.

This means that if someone gets your brainteaser right in an interview, there's, say, a 90% chance that they've heard a question just like it before (or one with a similar "trick"). The false positive rate is too high. Many people know the answers to brainteasers far beyond their ability.

If you were to ask a completely new brainteaser, as difficult as the canonical ones, but requiring a different trick, you would get a very low success rate.

Side note: there's too much criticism on HN about "what does this question have to do with my day-to-day job". This is a bullshit point. The point of a brainteaser is to test problem solving ability, creativity, etc. It may not do a very good job of that, but just because you won't be "reversing sentences in place" in your job doesn't mean that the question is invalid. Think about what the question is trying to test.

If they don't have a strong open source background (not everyone does), earlier in the process it helps to ask a couple very simple programming tasks, such as writing a function to reverse a string in place or the infamous fizzbuzz. It will weed out people with great resumes but poor skill.

    The best interview is a pair programming session

So once again, we're back to: there is no single best-way to fully assess a candidate.

Even for juniors and new grads, at school people code next to each other all the time.

    sitting next to someone while they code 

This is a terrible idea. If they come up with a brilliant solution can you actually use this in production? Who owns the code they write during an interview?

The last time I did anything like that was over 10 years ago. How would that skew your hiring process?

I have pair programmed (over 10 years ago) and I never saw any benefit from it. I work in an environment where every line of code checked into the repository is reviewed and documented in agonizing detail (welcome to the wonderful world of Medical Device software!) and I completely fail to see the benefit of pair programming.

But actually creating new code with someone sitting there - that's something that's not something I've seen often

Exercise: give them a class with empty functions, and some unit tests. Ask them to write the class and add some more tests. Also, one of the tests you provide must be subtly buggy.

I had an interview once that had me complete 3 small tasks on my own: a debugging task, a database task, and a small web site. All three had to be completed in under 2 hours. I wasn't given much direction, other than the specs of the 3 tasks.

While I think this was a moderate evaluation of my skills, it ended up being more a test of if I could complete the tasks on time, vs. the quality of my work: I ended up not making it to the next step of the interview because I was too concerned about quality code over actually completing the tasks (with mediocre code), and didn't complete the tasks, thinking that they would rather get a feel for my quality vs. if I can complete a task. I did it remotely, so they never really got a feel for my character or personality.

Had it been in a paired session, things likely would have gone differently. However, I did learn more about where their priorities were, and ultimately feel I probably would not have been a good fit.

He's obviously very talented at design (not necessarily only UI design but general program structure and framework design). But if you needed someone to write packet routing algorithms or design a high-availability database engine from scratch, that may not be his cup of tea.

OTOH, I bet there are developers who love brain teasers and puzzles who may very well excel at that type of work but couldn't implement a huge software system with many moving parts without having it collapse under its weight.

This a thousand times over. Go through their github (or whatever) profile, look at their commits, see how they write tests, etc. It doesn't take a huge amount of time, if you are any good at what you do, you'll quickly be able to filter out a failure (like, he doesn't have code that you can browse).

Not through any fault of that though, I realize that having a github account with projects and such that you work on is a great help, and I accept that myself not having one yet is a great disadvantage. What do you recommend to get started out on github? Is helping out open source programs still viable? Or now it seems either writing self-run projects, or forking others projects and helping them with those. Is there any clear starting point for githubbing? (and how do you squeeze it in after a 10 hour work day!)

Just questions from someone worried about my empty github profile :)

On the one hand I'd agree for students/juniors it isn't a deal breaker. But students/juniors that do have it are showing a great deal of enthusiasm and passion for programming.

Gitguy: "I used EC2, Blazboo and Chingbang to create an HA job queue that will never fail! It uses counting Bloom filters and I wrote it in Brainfuck."

Me: "What do you use it for?"

Gitguy: "Sending password reset emails."

These Rube Goldberg wannabes are generally more trouble than they're worth. You end up with a system that's a technological pastiche. The drawback is apparent when you try to hire new teammates. It turns out you can't find someone who knows the 12 esoteric packages your business is running on.

That, to me, is 100x more important than "how many zeros are in 100 factorial"

real engineers know that most good code isn't on github and never will be

Irrelevant. Fact is, having code out there makes it easier for prospective employers to get a feel for you, there's no way around that. No one said it had to be the best code or all the code.

Now, if you are working your ass off on proprietary code and write absolutely no code outside of your work, then you won't have it. That's understandable, but doesn't change the fact that it makes it harder for someone to figure out what you can do.

Actually that wouldn't be the worst idea in the world.

I agree with the latter. This is how I used to hire programmers, one month on a freelance contract first with a single project. The importance of the code itself is also overrated IMHO. It's just one tiny part of someone you add to your team, what about creativity, taking initiative, never giving up, being a team player, etc. Each of them are equally important to plain coding skills.

I've got a family. I need stability... and health insurance. Unless it's my only offer, a substantially better offer, or a dream job, I'm inclined to go elsewhere.

If I'm job hunting, I've got other offers on the table. Ones that do guarantee a salary for more than a month. Ones that include benefits. Ones that expire long before your trial hire would be over.

If you try using this method, I don't know of a top notch programmer that you would be able to attract, simply because the hiring process is too painful.

You can't really expect someone to quit their existing job just for a chance to work with you.

As a prospective employee wouldn't you like to kick the tires before making the leap into a new job that you might find yourself wanting out of six weeks in? There's so many things you just can't know about a company until you're on the inside.

Edit: Remember that 37s has a lot to offer: Remote work with great people, top-tier name recognition in the tech world, and a long history of building and releasing open source tools. That's a really compelling package compared to some of the bigger names in tech.

No matter how you look at it, you're asking me to take on significantly more risk with this process.

For developers that are in demand, it's a tough sell.

I had to deal with a crisis this week that boiled down to a poorly optimized SQL query or two. Of course, the manner in which they failed meant our DB started disk swapping and hurting our availability.

Contract to hire is something we do but it is rare. An initial try out period doesn't do us much good since the learning curve is long.

I don't trust take home assignments unless their solutions can't be googled. And it's a lot of work to create and evaluate problems of that nature. You want them just long enough, not a huge time sink for you or the candidate. You want them to demand a range of solutions so there are many ways to fail and succeed.

I agree we should not be doing riddles/puzzles. For me the most straightforward way to gauge a candidate's ability in an hour is to ask them to program on the spot and see how they act. Yes, it's a game and the candidate can learn to beat the system and oversell themselves. But the vast majority of people I don't want to hire do not go to these lengths.

The problem with this post is that it puts "puzzles", "API quizzes" and "math riddles" into one category. I can see how one can be offended by an API quiz such as "what does X do in Y?", especially if they have a track record in Y or could just look it up on the Web if they didn't know. But math riddles? Puzzles? I'd actually be grateful that I don't have to deal with technical catches and can purely focus on a solution.

One thing to consider is that recruiting, especially at a larger scale, is about minimizing false positives rather than maximizing the collective value. Sure, someone good at puzzles might turn out to be an average engineer but a bad one? I don't think so. Whereas, I can see how someone with a huge pile of code on GitHub can get stuck in a critical situation where if they write unit tests or how they design their APIs is meaningless.

I haven't really used their products but in terms of technical sophistication they probably aren't much advanced beyond CRUD type web apps (I may be wrong here).

If you wanted to hire a games engine developer or someone to design algorithms for google then brainteaser problems might be more relevant.

I personally detest abstract puzzles like "why are manhole covers round?" or (worse) "How do you move Mount Fuji?" AFAIK these types of questions are not and have never been used in engineering hiring at Google (sales, etc I know nothing about). WSJ and other sources have made this (AFAIK inaccurate) claim. It's possible individual interviewers do this.

My own philosophy is that a simple whiteboard coding problem is an incredibly useful litmus test. I don't mean anything incredibly complicated either. If you can't write a function that gives me a row of Pascall's triangle, that's a problem. Note: I don't expect you to know what Pascall's Triangle is, merely implement it when explained.

Some of you may think such a test is a waste of time but I assure you it is not. It's astonishing how many people can't do this (and even more astonishing how many that do make it through resume and even phone screens).

The key here is "simple". One problem is that interviewers fall into the trap of thinking these problems are too simple and make the problems increasingly harder. Worse, they may get in a situation of asking someone a problem that is either something they know (because they've covered it before) or they don't (where it's not something you can simply nut out). This is a useless indicator in my experience.

Just because you can create a function to return a row of Pascal's triangle doesn't mean you're a great programmer but if you can't, it almost certainly means you aren't. It's a negative signal filter, nothing more, but an incredibly quick and useful one.

API pop quizzes I'm not a big fan of either, generally speaking, but on the other hand if you've used a language sufficiently long, basic questions about commonly used libraries aren't unreasonable either.

Interviewing is a numbers game. Your goal is to come up with a process that balances time spent by the interviewer with finding a suitable candidate. Note that I didn't say accurately assessing each and every candidate. It's sufficient for the employer to simply find one qualified candidate even if you falsely determine someone who is qualified isn't.

A certain error rate is to be expected. It reaches a point where the time investment to reduce it outweighs the benefit of higher accuracy.

One final note: I detest (both as an interviewer and an interviewee) any kind of "take home" test (other comments have mentioned this).

As an interviewer, it's extra work to assess, people cheat, good candidates won't bother and so on.

As an interviewee, I'm not going to spend hours on you without speaking to someone first to find out how likely it is that you're a good match for me and the likelihood that you believe I'm a good match for you.

I would argue that a take home test is significantly more effort than a simple coding problem that doesn't have a significant increase in accuracy.

The one exception to this is automated testing where the problems themselves are relatively interesting. Some of the Facebook Puzzles fell into this category (and some don't). People may do those anyway for kicks. But again, there's nothing stopping someone going to Github and copying and pasting someone else's solution so what value is the filter, really?

EDIT: clarified my Pascall's Triangle comment. This isn't a trivia problem. I don't expect people to know what it is, merely implement it when explained.

I have over 10 years of programming experience and I probably would have miserably failed your "litmus test", especially if it were in an interview situation in front of a whiteboard. Maybe this just means I'm a bad programmer, but I tend to think that problems like "Pascal's Triangle" don't represent what we work on day-to-day.

I was astonished at the 25 minutes you claimed this took you, so I did it myself. It took me about two minutes to write this, which worked the first time I tried it:

    #!/usr/bin/env python

    def pascal(i):
        if i == 0:
            return [1]
        else:
            L = pascal(i-1)
            ret = [1]
            for i in xrange(len(L)-1):
                ret.append(L[i] + L[i+1])
            ret.append(1)
            return ret
    
    if __name__ == '__main__':
        import sys
        print pascal(int(sys.argv[1]))

If you program for a living, ask yourself in the last 5 years how many times did you have to write Pascal's triangle? How many times did you have to solve a brain teaser at work? I'll answer for me -- 0 times.

So I am advocate of posing a relevant problem and taking the candidate on the whiteboard through that problem. Not "let's imagine you have 100 horses, now one dies, blah blah..." but "Here is what we are working on. We have 1000 concurrent users, there is an open TCP connection to each, help us solve this particular latency problem". Yeah they might not know about Pascal's triangle but they if they can understand and show an ability to solve your particular problem then they are a candidate for you.

I honestly had forgotten the definition of it (the last time I saw it was in high school algebra), but I was able to get a working solution in about 10 minutes, including looking at the definition on Wikipedia. I'd say I'm a rather average coder (I'm certainly surrounded by people far better than I am, in any case).

If you had never seen or heard of Pascal's triangle before, I'd say maybe 20 minutes to code it would be reasonable.

However, this doesn't highlight good engineers. You know why? Because in the real world finding out the problem is many times a lot harder than coming up with a solution.

Here's a sample:

If I say to you that "I want these and these people classified / sorted, such that I want to find the people that have the best probability of being a good hire" ... I haven't actually told you the problem. I don't really know the problem myself. All I know is maybe that the hiring process is broken for some reason and I'd like to fix it, but I don't know the reasons (I can't put my finger on it).

And that's not a properly defined problem. We don't even know how to measure properly if a hire was good or not. And in the real world, that's how most problems are described to engineers - i.e. I have this pain, solve it somehow.

It's not intended to. It's highlighting bad engineers, not good ones. It's a negative filter, not a positive one.

> We don't even know how to measure properly if a hire was good or not.

Your company doesn't do performance reviews for employees?

Every performance review I ever attended was a bad joke, for the same reasons companies have problems filtering between candidates.

Seriously, if there's a problem with the hiring process (it was just an example btw, I'm not saying there is one at your company), then why do you think performance reviews work out well?

No, it certainly doesn't. I've never had to write Pascal's triangle, for sure, and I doubt I ever will need to. But on a daily basis I'm solving myriad little problems just like Pascal's triangle. Complicated? Of course not, especially with the libraries my programming environment makes available. But it's the bread and butter of what a software engineer does a daily basis.

If your objection is to make any sense at all, you must be contending that there exist--in the real world--people who can effectively implement solutions for complex problems but who can't code up Pascal's triangle on a whiteboard in a few short minutes. Where are these people? Pascal's triangle isn't a positive filter. It's not like we're hiring anyone who can implement it. It's a negative filter: if you can't do it, why should I believe that you can solve the much more complex problems software engineers regularly face?

Assuming you terminated the interview first, I'd go home and come to resent you for asking such a deceptive question. A simple coding test does not invite someone to conceptualize a solution to an unknown problem in terms of rows when the interviewer expects a recursive solution. I'd find solace in noting that the recursive solution is far less efficient than simply building the table manually (pop quiz: calculate exactly how much less efficient in 30 seconds!), and assume that you didn't realize this or you would have asked a more clever brainteaser.

Either way, I'd leave the interview irritated you didn't want to talk about the real problems you needed to solve and upset you didn't look at any of the code I made available before you scheduled our interview. I'd eventually feel grateful things didn't work out, but I'd steer friends away from working with your company, out of the conviction the people interviewing and managing the programming staff don't understand how to judge programmer value, and that it isn't a good place for people who get things done.

How's this for simple terms? http://en.wikipedia.org/wiki/File:PascalTriangleAnimated2.gi... It's on Wikipedia's page for Pascal's triangle. If your response to that page is to have your eyes 'glaze over' to the point where you can't find the animation (which has a simply-worded caption explaining the triangle), why should someone want to hire you?

I'm going to go for coffee, but there's a whiteboard right there. Shouldn't take more than a minute or two.

You would be right in deciding that. Over the years I've seen all these puzzle and math kids have the least productivity in delivering business software.

And besides if you spend say 30 minutes a day reading these interview puzzles online, you can game the system and get your way through. Without you even being tested on some basic skills required to win in the real world.

You may know all the algorithms from the book by heart. But what's the whole point if you don't how to use a couple of unix text processing utilities? Puzzle kids generally answer that by saying something like sed can be learn't by reading a manual.

If you can learn and work with sed by reading a manual on the job, then you can also learn and work with algorithms by reading manuals on the job too!

There is a fine line between brain teasers and the difficult problems that are our bread and butter. To answer your question: I would categorize many particularly subtle problems I've solved at work as brain teasers, so my number is high.

Relevant problems are a great way to judge someone's problem solving skills, but any problem that gives a glimpse into the way a person thinks is sufficient. I am probably not willing to give a candidate the amount of context to describe, let alone time to solve, the exact problem that I happen to be in the middle of on the day of an interview, but rather one we have already solved and determined to be interview-worthy. This means I certainly don't care about the solution, but rather the path to the solution. If you believe, as I do, that people tend to walk similar paths through every problem they solve, and that the path is important and the solution worthless, then the relevance of the problem is irrelevant.

That is definitely not what 99.999% of the programmers do every day. Professors in colleges, may be. But not programmers.

As for coding Pascal's triangle? It took me <30 seconds to Google it and find code I could copy/paste.

To be honest, seeing about 200 lines that looked like this:

    typedef struct Abc {
        Abc *n;
        Abc *p;
        void *d;
    } Abc;

    Abc *foo(Abc *p, void *x)
    {
        Abc *n;

        n = malloc(sizeof(Abc*));
        n->p = NULL;
        n->n = p;
        n->d = x;
        p->p = n;
        return p;
    }

Sorting it out was a bit of a challenge. I think the specific question that I was looking at deobfuscated into reading words and definitions in from the command line and maintaining them in a sorted list, then looking up the words and printing their definitions.

And I agree, it was a great interview question. I even had the ability to sit in front of the computer and test my fixes to make sure it worked.

1. You need to malloc 3 words, and you're only allocating one (you want sizeof(Abc)) 2. Abc in the struct will not resolve (change to struct _Abc and the struct name to _Abc) 3. You should cast the result of malloc/check for failure 4. This is bad memory management practice, I think.

2) No, 'struct Foo' is in a different namespace from the typedef'ed name. 'typedef struct Foo Foo' is perfectly valid.

3) Yes, absolutely.

4) Not sure what you mean.

Here's a hint: You'll likely miss one of them unless you realize what the algorithm is trying to do. It's an algorithmic error. The other one is findable from looking at the code without context, but the correct fix isn't entirely obvious without realizing the way the code works.

Pretty good, though :)

I think function foo is supposed to insert a new node (n) into the list before node p, but it's not done properly (while n points to p and p points to n, n does not point back to the item before p).

That should be enough for you to figure the last 2 errors, in any case.

n->p = p->p

I don't see the last one - unless foo should be returning n?

     prepend(NULL, &something);

     if (p != NULL)
         p->p = n;

I don't think I'd expect everyone to get every bug without a little bit of prodding in the right direction. Being able to step through the code and show what it's doing to the data structures would be a first step that would make me happy. Recognizing that the data structure looked like it's a linked list, and finding and and fixing a couple of the bugs, and I'd be very happy.

Unfortunately C++ requires the cast.

Realizing and taking advantage of this property of Pascal's Triangle significantly simplifies the problem. I guess it should be obvious, but I didn't consider using it. I don't think it's because I'm stupid (though I might be, I'm not sure) - I think it's because these kinds of problems are far removed from what I work on day-to-day.

Exactly! The OP says he "clarified" the problem, but I can clarify tons of math problems for you where the solution is staring you right in the face yet its impossible to solve if you haven't seen it previously.

example:

We all know 2,3,5,7,11,... are the primes

So the next guy is 13.

We all know 4,9,16,25,36 are the squares

So the next guy is 49

We all know 1,1,2,3,5 are the fibonacci numbers

So the next guy is 8

..

..

..

..

..

So, what's the next guy in 1,5,7, 15, 20 ?

( Remember the solution is right in front of you, but 90% of people on average won't figure it out. These are called convolution series. The solution is 28, if you haven't figured it out yet. )

(edit: provided solution and fixed a simple numerical error: dekz, thank you for spotting that )

What sequence is this and what is the justification for 28 being the next number?

[1] http://oeis.org/

(1) correct an error in the question which renders it unsolvable (a_3 should be 9, not 7)

(2) guess that the invented term "convolution series" doesn't have anything to do with the mathematical concept of convolution[1]

(3) a_n = nth square number minus nth prime minus nth Fibonacci number.

Staring you right in the face!

[1] https://en.wikipedia.org/wiki/Convolution#Definition

Closest I can come is square minus the other two which gives:

1 5 9 15 20 28...

Yep, you got it! That is the solution. There is a method to the madness. Given series S squares, F fibonaccis, P primes, with an op of minus, the convolution series

C = S op F op P = S op P op F

gives you 1 5 9 15 20 28... ( ps the third term is 9 and not 7 as stated in the original problem. In my defense, it was 7pm and I was drowsy when I posted that, so I must have looked at my watch, seen the 7 on the dial and then wrote 7 instead of 9. Apologies )

In any case, even if the candidate didn't figure out the convolution series, but vocally tried calculating differences or tried to look at the numbers in a different base than 10 or even complained that she can cook up an arbitrary rule for the remainder of the series, that kind of thought process is certainly interesting and playful interview material .... if the post calls to a keen eye for patterns. The key is that the answer is not what you're interested in, ever. It's always the thought process.

2^2, 3^2, 5^2, 6^2, 7^2

4^2? Did I miss something there?

Are you seriously contending that Pascal's triangle is such a problem?

We all know that "Aha!" problems exist, that there are algorithms that are simple in hindsight despite being anything but obvious before we were told the answer. I don't think anyone here is proposing that such questions are good for interviews. But do you really believe that producing rows of Pascal's triangle, when the triangle has already been explained, is actually one of those "Aha!" questions?

If you aren't contending that, I'm not sure what point you're trying to make above.

Once one understands the actual problem (wikipedia has a rather nice graphical pyramid showing a few rows being generated), it should be a rather simple task to implement a naive version of it in any modern language. Is my ruby version the best performing? Of course not. However, it demonstrates some obvious insights into it such that the first and last members of a row are always 1's.

If I were hiring engineers, I'd be slightly wary if they couldn't at least make basic insights such as that and get darn close within a few minutes. Obviously, there are nerves and other things that can cause issues during an interview process - but one should arrive at a general solution rather quickly, even if it's not the most beautiful.

I don't think anyone is trying to be condescending - but the quick naive approach using an array and calculating some sort of offset to get your values is simply bread and butter in our profession.

   def pascal(i):
        o=[1]
        for j in xrange(1,i):
           o.append(o[-1] * (i - j) / j)
        return o

Some of the best interviewers I've seen approach interviewing almost like teaching, in that they'll ask you questions that lead you towards the answer, just to keep the conversation flowing so they can see how (and whether) you think.

For instance, in this case, after providing a correct recursive implementation you'd hopefully add the caveat that it might not be the best way to do it because of maximum stack depth, and you could talk about that a bit, as well as workarounds. You'd likely be able to easily offer an improved version that iteratively updated a (properly sized) row in-place, [1,0,0,...,0] -> [1,1,0,...,0] -> [1,2,1,...,0], etc., until the last digit became 1, in which case you're done.

Then you'd have a conversation about the time and space complexity of that method, you'd hopefully be able to figure out that it's O(N^2) in time and O(N) in space, and the interviewer would ask you whether you could do better. You'd tell him that it's optimal re: space, but that there could possibly be an implementation in O(N) time, and then you'd think for a moment.

From there, the interviewer would give you a gentle prod (after all, this isn't a math interview and we don't have all day) by suggesting you look at the ratio between neighboring elements, and then it's easy to, say, look at the eighth row and see that the ratios are 8/1, 7/2, 6/3, 5/4, 4/5, 3/6, 2/7, 1/8 and figure out dspeyer's formula from there.

The nice thing about interviews in this format are that they don't require any specific prior knowledge; in fact, they work much better when the candidate has never seen the problem before, because you actually get to talk through it with them for the first time.

I realize that algorithms aren't everything, but at a place like Google, at least, they're a lot more important than at a typical Java-shop, so I think questions like this are perfectly fair. If someone can't make it through an interview like that, they're going to have a lot more trouble on the job when they need to cast a complicated machine learning algorithm that's just been invented by the guy down the hall into map-reduce form and get it running well enough so that it can run in real time and actually be tested...

I was meaning to comment on how dspeyer was portraying his solution as a simple iterative alternative to jemfinch's recursive one, when in fact it was an entirely different algorithm that calculated the results in a different way. I thought it was strange that he did not mention this.

For an algorithms heavy job, I would hope a candidate would be able to derive this equation with a little guidance (ie, reminding them about n choose k and how to write that with factorials)

(Were one to ever write such code in production, he should be escorted from the building immediately!)

    from itertools import tee, izip, chain
    pascal = lambda rows: reduce(lambda acc,n: acc+[(lambda it: [sum(x) for x in (lambda it,n=2: izip(*((lambda it,n: ([next(it) for _ in xrange(n)],it)[-1])(it,pos) for pos,it in enumerate(tee(it,n)))))(chain((0,),it,(0,)))])(acc[-1])], xrange(rows), [[1]])

    num_rows = 10
    print '\n'.join( ' '.join(map(str,row)) for row in pascal(num_rows) )

    (define (pascal n)
      (if (= n 0) (list 1)
       (begin
       (let ((L (pascal (- n 1)))
           (ret (list 1))) 
       (do ((i 0 (+ i 1)))
           ((= i (- (length L) 1)))
           (begin (set! ret (append ret (list (+ (list-ref L    i) (list-ref L (+ i 1))))))))
       (append ret (list 1))
        ))))

This is better scheme:

(define (pascal n) (if (= n 0) (list 1) (let ((L (pascal (- n 1)))) (append (list 1) (map + (cdr L) L) (list 1)))))

As for your particular questions:

1. map yields an error under mzscheme

I'm at work, and didn't have access to a proper scheme, so I used an online REPL[1], which apparently is more forgiving. Regardless, you can write your own map (a good exercise!) that stops as soon as one of the arguments is exhausted. Make it tail-recursive[2] as well!

2. list-ref is bad

Lists are beautiful data structures designed for recursive algorithms. If you are using lists and not using recursion (or a hidden recursion in the form of map), you're doing something wrong. list-ref uses list as a vector, which has a performance implications - your algorithm is O(n^3) while mine is O(n^2).

3. set! is bad

Margins are too small for a proper explanation :), but basically set! has side-effects[3], and functional programming should avoid having them.

Have fun learning Scheme!

[0] http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-9.html#... [1] http://www.bluishcoder.co.nz/jsscheme/ [2] http://en.wikipedia.org/wiki/Tail_call [3] http://en.wikipedia.org/wiki/Side_effect_(computer_science)

(define next (lambda (l) (if (null? (cdr l)) '(1) (cons (+ (car l) (cadr l)) (next (cdr l))))))

(define pascal (lambda (n) (cond ((< n 1) '()) ((= n 1) '(1)) (else (cons 1 (next (pascal (- n 1))))))))

Here's mine:

  #include <stdio.h>
  #include <stdlib.h>

  long *pascal(int row) {
    long *ret = (row == 1 ? NULL : pascal(row - 1));
    ret = realloc(ret, sizeof(long) * row);
    ret[row - 1] = 1;
    for (int i = row - 2; i > 0; i--)
      ret[i] += ret[i - 1];
    return ret;
  }

  int main(int argc, char *argv[]) {
    int row = atoi(argv[1]);
    long *data = pascal(row);
    for (int i = 0; i < row; i++)
      printf("%ld ", data[i]);
    printf("\n");
    free(data);
    return 0;
  }

I ran your code and it does run well. However fails for large cases such as pascal(200000).

You mentioned you worked for Google and Amazon, how can you modify the code to get an answer for pascal(200000)

  1. use double to get approximately-correct answers
  2. use an abitrary-precision library like GMP

  #include <stdio.h>
  #include <stdlib.h>

  int main(int argc, char *argv[]) {
    int row = atoi(argv[1]);
    long data[row];
    for (int i = 0; i < row; i++) {
      data[i] = 1;
      for (int j = i - 1; j > 0; j--) data[j] += data[j - 1];
    }
    for (int i = 0; i < row; i++) printf("%ld ", data[i]);
    printf("\n");
    return 0;
  }

So far dspeyer, has one of the most efficient algorithms.

  def pascal(i):
        o=[1]
        for j in xrange(1,i):
           o.append(o[-1] * (i - j) / j)
        return o

  def pascal2(i)
    o=[1]
    i += 1
    for j in 1.upto(i - 1 )
       o << o[-1] * (i - j)/j
    end
    o
   end

  def pascal(i)
   o=[1]
   i += 1
   if i == 1 then
    return o 
   end
   i.even? ? d = i / 2  - 1 : d = i / 2 
   for j in 1.upto(d )
            o << o[-1] * (i - j)/j
   end
   o[0..i/2] + o[0..i/2 - 1].reverse
   end

  def timeA(m,n)
    begin_time = Time.now
    send(m,n)
    end_time = Time.now
    p "Time elapased #{(end_time - begin_time)*1000}    milliseconds"
    end

  ruby-1.9.2-p290 :037 > timeA('pascal2',40000)
  "Time elapased 949.9490000000001 milliseconds"
   => "Time elapased 949.9490000000001 milliseconds" 
  ruby-1.9.2-p290 :038 > timeA('pascal',40000)
  "Time elapased 470.48699999999997 milliseconds"
   => "Time elapased 470.48699999999997 milliseconds" 
  ruby-1.9.2-p290 :039 > timeA('pascal',80000)
  "Time elapased 1776.224 milliseconds"
   => "Time elapased 1776.224 milliseconds" 
  ruby-1.9.2-p290 :040 > timeA('pascal2',80000)
  "Time elapased 5722.419000000001 milliseconds"
   => "Time elapased 5722.419000000001 milliseconds" 
  ruby-1.9.2-p290 :041 > timeA('pascal',200000)
  "Time elapased 160416.34199999998 milliseconds"
   => "Time elapased 160416.34199999998 milliseconds" 
  ruby-1.9.2-p290 :042 > timeA('pascal2',200000)
  "Time elapased 235113.38 milliseconds"
  => "Time elapased 235113.38 milliseconds"

But that is under the (large) assumption that a suitable recursive algorithm has been found. The programming or coding is not the issue here. It's understanding the problem well enough and then identifying a recursive strategy.

    getRow = (n) ->
        rows = [[1]]
        for i in [1...n]
            prev = rows[i-1]
            rows[i] = []
            for j in [0..i]
                rows[i][j] = (prev[j-1] or 0) + (prev[j] or 0)
        rows[n-1]

    (n) -> [[1]..n].reduce (p, c, i) -> ~~p[j-1] + ~~p[j] for j in [0..i]

That's about 9 seconds per line.

Unless you are typing a program that you memorized - you cannot get such speed.

Am i a bad programmer now if i cant get a running program in less than 5 minutes ?

I would rather let people apply with code they are proud of like in every design job than solving litmus tests/puzzles

There are plenty of software jobs where that statement is absolutely true. I am not interviewing people for those jobs.

Taking a well-defined problem such as "write code to print the Nth row of pascal's triangle" is substantially easier than most of what we deal with on a day to day basis. The hard part is discovering and defining the problem well enough to get the point where it is as easy as "here is what we need to do, write some code to do it". Once you get there, whipping out the code needs to be second nature.

As cletus said in his post - a problem like this is a negative filter. If someone has a hard time whipping out the code for a well-defined problem like that, they will have a very hard time here.

If they can't, it doesn't mean they are a bad programmer. It doesn't mean they can't be successful in the industry.

It is simply a strong signal that they would not be successful at Google. And since I'm interviewing people for jobs at Google that's what I care about :-)

Would you rather be tested on your ability to comprehend a multi-kloc codebase and make correctness-preserving modifications to it?

That depends on how rigorous and well-maintained the existing test suite is, or even worse, that it is only half done. Which half? Guess, I think you'll be surprised.

Solving Pascal's Triangle is a straightforward yet not-completely-trivial logic exercise. It's the kind of algorithm design task that "serious programmers"[1] need to do, successfully, every day. Goofed logic bugs in algorithms are Very Expensive bugs. People who can avoid them are very desirable. This particular problem can be solved in just a few lines of code with no edge cases, which makes it a great candidate for a test to detect those developers.

[1] And yes, of course one can be a "successful software engineer" without being a "serious programmer". Writing CRUD logic and the like may be its own skill (and one not helped by algorithmic reasoning), but CRUD developers are cheap and plentiful. Tests like this give you a marker for the people who are hard to find.

By the same token, you appear to be assuming that successful candidates aren't. But I'm not trying to polarize.

Wouldn't it be better to invent a scenario for which a solution/algo must be constructed, during which the interviewer and interviewee can bounce their relative expertises? Is it too much to ask for the questions not to be rote(-ish) Pascal's Triangle stuff?

Talking about software design methodology cannot tell me whether someone can write new algorithms correctly. If I have a job that requires people to correctly design algorithms, I think asking questions about algorithm design is entirely appropriate.

Are you seriously suggesting that anyone has code for a line of pascal's triangle memorized? Why would they? Just in case they got asked for it in an interview?

This is one of the advantages of this over something like quicksort: it's obscure enough that the candidate won't be able to recite it.

Not only did I find this very helpful in evaluating prospective employees, several people that we hired told me later that it was the most fun they'd ever had in an interview. That is, of course, a biased sample; some of the people that I interviewed obviously did not find my interview style entertaining.

The litmus test that I generally used was whether they were able to, given an erroneous output of their own creation, they could reason about why it happened and fix it before the hour-long interview slot was over. Also, if I (watching over their shoulder) couldn't spot the cause of the bug, I considered that it was probably unreasonable to expect them to be able to find and correct it during the interview.

While I encountered a lot of candidates that froze in the face of compiler errors or started making changes without stopping to think about what the cause of the error might be, I never once got the feeling that anyone was less familiar with the code that they submitted than someone who had written it themselves.

For 90% of engineering tasks, you're just plugging together different technologies other people put together. Oh hey yeah, Unix + HTTP + Rails + Javascript + Some Dude's API.

The amount of SHEER INTELLIGENCE necessary to untangle the above is, I think, not very high. It's a low threshold of competence. So, for that range of problems, fumbling around for fifteen minutes in front of a whiteboard will display the necessary grasp of logic.

The hard part in interviewing (and what people usually screw up), in my opinion, is almost entirely finding someone with a can do attitude who you can get along with for 40hrs every week.

Almost anyone can string stuff together; finding people that you can trust is way harder.

You do not have to create perfect code on the whiteboard. You just need to demonstrate that you can code. Sounds like you would have been fine.

You are right! In fact I seriously doubt any serious software project that is solving business problems ever works on these sort of problems.

These problems are basically more of testing mathematical knowledge through programming.

They may make a great interview to hire professors but not programmers.

https://gist.github.com/1567842

I tend to like to rough code out Lispishly, but if I'm actually running stuff I'll often do it in C.

I've developed high-performance medical imaging applications. I've build large distributed financial systems. I've built 2D/3D games. I've done a ton of stuff. Hell, I'm a software engineering hiring manager at my company. Yet I have no idea what a Pascall's triangle is, or how to give you a row of them.

I consider questions like that to be as useless as brainteasers in determining whether or not to hire someone. Sure, it can help give clues about a candidate, but if you're seriously considering rejecting a candidate on that point alone, I think you've got a problem in your hiring process.

Pascal's triangle is simple a triangle which starts at 1 and where each number in the next row is the sum of the two numbers above it, with 0 being assumed for numbers on the edge.

A program to produce one is nothing more than a couple for-loops, printf, and some basic logic.

All I'm saying is that there needs to be a multi-faceted look at a candidate. Some candidates are going to be so obviously poor that they do merit instant rejection. However, in my experience, that is the exception rather than the rule.

And yes - trivial programming questions do weed out a fair number of candidates. However... it's tricky. A trivial programming question can look different to many different people, especially if they have varying skill levels and depending on their level of comfort in interviews.

I try to give people the benefit of the doubt. If one question stumps them, I'll try to gauge their understanding with another similar (but different enough) question. If that doesn't work... sometimes I'll give them another shot, but usually I'll start winding the interview down.

I would ask you to explain, you would smirk inside, I will be rejected.

I am sure google should have enough data by now to see if there is correlation between knowing this problem and performance of hired programmer. I will be suprized if it has any relation.