2. a programming task. Choose 1 of 5 tasks. The tasks are not abstract problems or puzzles, but come out of the designs we've implemented. We ask for 100 lines of code and not more than 2 hours. They take as much calendar time in days as they need, most have full time jobs, and let us know when you're done. The candidate then comes back, typically in 2 to 7 days and hosts a code review in front of our team. I give strict instructions before hand: The purpose of the review is to learn how they thought through the problem and how they solved it, not to criticize their style and approach.

3. If we decide we like them to this point, the third interview is with managers from other functions. How well does the candidate communicate, come across to non-developers, express interest in the company and role, etc. It's a check point to look for concerning weaknesses, as well as get buy in from the broader organization.

We like this approach because it allows the candidate to code in a much more natural environment. They can take the time they need and comfortably solve. No one spends their professional career coding on a white board. This approach so far has yielded excellent results. We ask developers how much time they took in the programming task and why they chose the one they solved. The programming task is telling, not in the quality of their code and how long they took, but how much did they get into it? We have had the range from candidates who did not complete it at all and opted out, to candidates who stumped 40 year veterans with elegant code. In every case, we learn how well they can express their thoughts, and importantly, their level of love for the discipline. This is as important to me as any other attribute.

Plus, if your company gives someone what they might think are ridiculous questions or they realize it's just not what they're good at, they can easily reject proceeding any farther with the interview. If someone walks into a crappy whiteboard interview, they feel kind of stuck. Seems to be better for both sides. Less pressure.

But any time take-home tasks come up on HN, there are always people here complaining about it and even demanding to be paid for their time.

I get that it doesn't suit everyone, but nothing does. Coming up with an approach that works for the company and candidates is really hard - as evidenced by the many, many threads on this subject here on HN.

Perhaps the best compromise would perhaps be to permit the candidate to perform the task there and then or take it home, but this may require more of a commitment from the interviewer, if they then need to additionally devise tasks to be completed in a shorter time.

That's not how developers spend the vast majority of their time. It's a poor signal in my experience. What it selects for is candidates who are immediately comfortable collaborating in real time with someone they have never met. That doesn't mean you aren't getting good candidates. What is means is that I spend far less than you finding good candidates because I also hire people who are great developers and who would fail at your interview.

The compromise here is to talk about the employee's past projects. The downside is now the interviewer needs to get up to speed quickly, and won't have all the necessary context of the organization and (potentially) industry. Not to mention the fact that candidates lie a lot about how much of a role they actually had in projects.

I think what gives our approach balance is that we are also committing to making a time investment. Reviews typically last an hour, which means that the 2 hours we ask from the candidate is matched with 3-5 hours of our engineers' time to review and discuss with them.

I somehow feel this rewards opinionated developers.

Also right now I'm looking at a couple places because their recruiters called me up and said they were willing to meet my requirements which are basically pay - I don't care about most of these things, if they want me to do stuff and are willing to pay my rate and if they, after looking over my cv believe I can handle their stack, that's it. I don't really have any questions for them.

So I guess you don't have people looking for people for you - you only have people coming in that think your company is special enough that they really really want to be there and are invested about finding out everything just to confirm that commitment on their part is good?

on edit: this of course supposes that 90% of the companies out there are very similar and do not have any problems that are so interesting that I should feel prompted to ask. If I ever applied to the Internet Archive I'm sure I would have things I would like to ask.

If I hire a developer "to build X," I would rather work with the developer who explains that X is the wrong thing and we should actually build Y, than the one who shrugs, asks no more questions, and builds a really excellent X - that later turns out to be the wrong thing to have built.

I don't want code that stumps anyone. I want code that delights 40-year veterans, for sure. But the signal of genuine elegance is "oh, that's a lovely way to do it" much more often than it's "how the hell does this work?".

That’s not to say I don’t think this process has promise, but I’d need to be convinced that the above two issues can be fairly dealt with first.

It sucks, but you only need to be better perceived than the next best competitor. Most folks don't have 40 hours to spend on a 2 hour max coding challenge. Hopefully the ones that do are already employed and not currently competing with you, or are so inexperienced that 40 hours produces horrific over-engineered abominations instead of clean efficient readable code.

If I want this job, I would spend this extra time to submit this better solution because obviously that will improve my chances of getting the job. But that means that these take home assignments might not be such a good indicator of efficient proficiency. You might be overvaluing people spending more time on these tasks and undervaluing people spending the allotted time on these tasks. For example, a better developer who did write a solid solution in under 2 hours might get passed over because her solution seems not as good as mine where I did spend another hour to rewrite my initial messy solution.

The goal is as much to learn how they think, solve problems and communicate as it is whether they know how to write software.

I'll give one example: One task is to figure out how long it takes to grab the current time from the operating system. We use a rather obscure OS. A candidate chose this and solved it simply (though it's got a couple of points to mind), but then he got curious and wanted to know whether a virtual machine had a different timing and then what the performance on a different operating system was. He told us that he spent about 8 hours altogether. I felt a bit bad on one hand, but on the other, it's exactly what we like to see: Solid programmers who love programming technology for its own sake.

I think it's common for an on-site interview to include a reasonable chunk of programming, and using a continuation of the take-home to do that is good because it means the candidate isn't starting from scratch. Coming in warmed up should minimise noise from nerves, unfamiliarity with tools or context, etc.

I recently had a humiliating experience. Just was approached on LinkedIn and at first they sent me a take home. I learned the front-end tech which was required for it, and then coded it. They just wanted me to fill out a doc but I put it all up on stackblitz. Another portion was back-end which I also put up on another online env so they could see not only the code but how everything was laid out.

Get to the over the screen share interview - just nervous and both the interviewers were impatient, and just the atmosphere was so nerve-wracking - they did nothing to make it more relaxed. Just seemed like the senior guy was stressed out and I was wasting his time. My brain kind of shut off and the seemingly simple problem at that time just seemed insurmountable because I felt myself second guessing my every thought.

Been developing for a number of years for a few companies, and I guess I haven't prepped for the tech interview - I thought my existing skills are what they need and should be enough to evaluate but boy was I wrong.

Needless to say they didn't go with me. It's not like I needed the job, or probably wouldn't have taken it anyway since what their maximum range was lower than what I make, but the startup seemed quite interesting. Just such a bad experience.

If they're not placating you at all than I doubt that'd be a great environment to work in. Or they'd already selected someone.

I think if I was interviewing using your process I would submit code because I would guaranteed a chance to talk about it with other developers.

Temporarily only

I guess the tasks are something like "take this scaffolding and implement thing x". What would you think if the candidate responded saying the scaffolding does not allow to implement X properly without sacrificing things y and z, and instead of turning in code for review would show up for a discussion about the compromises and deficiencies in the scaffolding?

Responded how, when? (At what point in the process)

> > instead of turning in code for review

I.e. something like instead of turning in code example they turn in a motivated analysis of why the scaffolding provided cannot support "proper" (and then analysis of so called proper) implementation of the thing asked, provide analysis for changes required in the scaffolding.

Basically, the candidate would turn "provide and review code" process into "let's talk about architecture, implementation and tradeoffs". Would you discard such a candidate for not following the process because you expect seniority or would higher level thinking be a plus?

Whilst the challenges were not particularly complex, you also had to ensure that the person who was going to run your code could stand up your environment, so part of the second challenge was "thoroughly document how to run your code, the environment needed, etc, etc."

Life is too short to deal with this kind of stonewalling.

I did one recently after the initial HR person chat. I spent a morning on it as they suggested 2 or 3 hours. But then I was rejected for the most pedantic reasons (2 white space PEP violations), not setting up mocks for testing (that would have doubled the scope of the project). Providing an HTML response rather than JSON when the spec asked for a "page, don't bother about styling". And various other trivialities.

I tried passing on my complaints, but the HR girls didn't like that. Pretty disrespectful after giving up a morning of my free time.

Sennder is the company, as I feel that we should be naming and shaming.

I don't see how mocks would have doubled the scope of the project. I think I have mocks set up in a few lines of code or so. If your project is so difficult to test or if it is difficult to set up mocks for it maybe there is a serious flaw in your project structure?

> Providing an HTML response rather than JSON when the spec asked for a "page, don't bother about styling".

I don't understand what you mean with this. Did you provide a HTML response or did you provide JSON? I did a super simple HTML response with no styling and that was fine. If you did JSON I could see how they would mark it as a mistake when they clearly asked for a HTML page.

The company was Goldman Sachs. Name and shame.

I don't interview much these days, but I spent several years of my career sifting through CVs and interviewing candidates.

The number of candidates with seemingly impressive CVs and apparent several years of experience, yet whom basically couldn't code, was shockingly high. And when I say "couldn't code", I'm talking "couldn't even fizz-buzz" levels of incompetence.

The fact is that there are many devs out there who coast through jobs, while all the time it's their colleagues who are doing all the work.

I wasted, many, many hours of my life before realising that it was pointless interviewing candidates that couldn't code - asking them to perform a sort of gatekeeper test improved things immensely.

So it might seem insulting to have to perform these entry tests, but unless you already have code publicly available (e.g. on GitHub), you shouldn't take it personally.

It seems you don't enjoy coding?

When I set it up the goal was to avoid a lot of the issues I've personally had or heard about with coding challenges: they take way too long, they're too abstract, or you feel like the company might just be using you to get a couple of days of free work on their actual codebase. For this reason I feel fine about not paying people, as it's just a small part of the application process and not free work for us.

I think it's worked pretty well so far. It happens after the initial phone screen and before the first technical interview. We send people the challenge, they return it whenever they want, and if we like it we set up a technical interview. Since the challenge uses our tech stack and is similar to the work we actually do, a large part of the first technical interview is discussing their solution. Why they chose certain patterns, why they added a certain library, how they'd consider testing it, why a certain function might be slow with 10k entities, and so on.

You don't normally pay candidates for the time they spend doing the whiteboard interview, do you? You can view the take-home task as a replacement of the whiteboarding session, equally with no pecuniary obligations for the candidate.

Then what? Would it feel good to pay

He asked me vague question, like what is architecture, I started to reply with what design tradeoffs I made on the app I was working on. He literally laughed at me, and said that is not architecture. I was shocked that someone would laugh at a candidate but said ok, what does architecture mean to you? He responded, I ask the questions here not you. I immediately ended the interview. Worst interview I have ever had. Every time I see that company brought up, I think back to that experience.

At some point the interviewer asks which I would choose in a project, C or C++ and reasons why. Basically answered that it depends on the project, but that both languages had their pros and cons, and it all depends. I told him I did not that many projects under my belt at the time.

Interviewer got somewhat pissed off and kept asking me about specific languages constructs which I was not yet familiar with. In a way felt like he was trying to ambush me and make me feel incompetent. I mentioned to him that I was not yet experienced enough to answer these questions, but was eager to learn. Well he did NOT like that either and ended the interview promptly.

It seems like the guy really did not like me (which is fine), but also wanted to turn this interview into some type of power trip ....

Had this been today, I would have politely declined and left. But I was a young and eager cub. Live and learn!

All jokes aside he was not foaming at the mouth or unpleasant when we first started the meeting, and I'm not the quickest one to pick up on subtle body language cues. So your guess is as good as mine.

Sounds like a good judgement thing to reply :)

Too many times we read someone's experience, which even if only half true, would be horrifying, yet no indication of who the company was.

Similar response.

Nobody has ever asked me how often we need to build calendaring software, because I explain right up front that while this is a “toy” question, our core product functionality schedules people, and nearly every feature from a calendar app has some analogue to things we either do, or are asked to do but haven’t prioritized yet.

I think it’s ok to ask “toy” questions, but I also think that there should be a ready answer to the question “Does this have anything at all to do with the job?”

p.s. We don’t ask a question directly about scheduling, for a simple reason: Almost everybody understands the basic idea of a calendar, so it’s a more “level playing field” for candidates to think about calendars than schedules.

Again, thanks so much for naming the guilty party (NoRedInk) in this case. So that we can firmly add them to our list of companies not to bother engaging with, if this is the kind of behavior we can expect from their interviewing team.

The above anecdotes speaks volumes about the mentality that drives companies like these -- and the "culture fit" they're apparently looking for.

I actually had the pleasure of interviewing at one of these companies before. They had a take-home project, which was to choose and implement and couple enhancements to a toy app. In the subsequent conversations, we discussed how I approached the problem, details of my design, technical tradeoffs, etc -- all the sorts of things you would expect a professional sw engineer to be able to do on the job. It was challenging but fair, and I was struck by how reasonable the process was. It left a wonderful impression on me.

Take-home projects are not a perfect solution to the interview problem. A major issue is that they require candidates to have free time outside of work to complete them. Personally, though, I'd much rather spend a few hours per application working on projects than countless hours prepping for stressful whiteboarding puzzles. As a developer who prefers to digest problems slowly, I look forward to the demise of on-the-spot whiteboarding interviews.

If you work at a company that subjects candidates to such high-stress algorithmic riddles, and you have the ability to make your process more sane, please consider following the lead of cos on this list.

Also many companies will insist on a take home project in addition to everything else, including a whiteboard interview. So you can pour time and effort into this project only to be eliminated further down the line (or just ghosted).

It's like they assume applicants are applying to one job at a time and have no other work or life commitments.

I dunno, it feels like the tech industry has forgotten how to interview candidates, as in "just chat with someone like a normal human being to see if they are a good fit for what you need", and like many other aspects of the profession have fallen for dumb processes, fads, cargo-culting, outsourcing to third parties, and dubious technical solutions in lieu of experience and social interaction. I learned this early in my career, and found I can pretty much size up someone's technical chops - assuming they're in my wheelhouse - and get a good idea of their professionalism - with a short, half hour phone/Zoom interview. Yes, I can see how this might lend itself to bias and prejudice, but it seems the solution there is to train interviewers to overcome biases and have more diversity (in all senses of the word) in your interviewing team.

Not everyone has the inclination or time to code outside of work. Many of these people are still very competent developers.

But frankly I don't really buy that excuse. Those people have time for multiple coding tests but they don't have time for a (small) public project? Ehhh. Not convinced. If you can get away with no portfolio to begin with then you're not going to be inclined to accept coding assignments.

I think the best approach would be giving the candidate a broken app and have they debug it live or in a take-home debug-and-fix or debug-and-improve exercise. Could also have some refactor assignment.

Has any company tried something like this before?

They lower the cost for the company to zero. So they can widen their pool, and take a chance on more candidates, with no extra cost incurred for them. If you take this to the extreme – every company asking every candidate to complete a take home – then the result is far worse for candidates.

There is no feedback mechanism for candidates during the process. I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable. On occasions where we had a system for capturing progress, I found that most candidates far exceeded the time recommended to them. Some were spending up to six hours. The frustrating thing was the extra time never made a difference to the outcome. I could tell from their initial hour whether they were going to pass.

As a hiring manager I've used them, but work hard to be considerate to the candidate. I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

When we get a take-home submission, we essentially run it through our code review process, and as one of the engineers who does reviews, it usually takes me nearly as long to thoroughly examine a homework solution and write up my feedback for internal discussion as it would to do an in-person interview.

Being able to review a candidate's code on my own schedule rather than being yanked from the middle of some knotty problem to go do an interview at a fixed time is a big quality-of-life win for me. But the rate limit isn't much different than it would be with regular interviews.

All that said, it's totally possible our process is super atypical.

With a 2 hour exam, there's an end to it, and you know how much time it should take to do the problems. Much better.

It's even worse if you're being graded on a curve, as you would be in a job interview. You're competing against others who will spend infinite time on it.

> I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

Why don't you just let candidates decide for themselves how much time they want to spend on the task? You can set the expectations, saying that you expect that the task of a given complexity will to take between 1 and 2 hours; but candidates should be free to spend as little or as much time on a task as they like. Some candidates will value their time and spend no more than is advised; others will be sufficiently obsessed by the task to spend more time. Why should the latter group feel guilty about exceeding the recommended time limit?

- Whereas if such an individual, because of their desperation or lack of demand, has enough time and motivation to elegantly solve the problem, then what's not to like?

I think a greater concern would be that with take-home assignments it's impossible to be sure you are actually evaluating the right individual. How can you be sure they didn't receive help from someone else?

Their desperation to get the job doesn't necessarily carry over to their day-to-day programming. If they take a month to do a week-long task, I would want to know.

In terms of cheating, that can be easily resolved by asking detailed questions about the candidate's submission. It is also helpful to show that you took the time to review it. (There's nothing as frustrating as taking a day for a take home test and getting no response).

Won't you see, in the code submission, at least some evidence of experience? The way the candidate organises their code; the overall style; the third-party libraries they choose to use (or not to use); the tests they write (or don't write) — doesn't it all, combined, speak of experience or of lack thereof?

There are still other stages of the interview, where you can assess candidate's experience more directly, by asking about what they worked on.

I agree that it's useful to know when a task that should have taken x long takes 5x or 10x; but it may be because the candidate set a higher bar for themselves.

I'm going totally nuts about this particular problem. I have written in as clear terms as possible that I really don't want the candidate to spend more than 2 hours, but I don't feel I am going to convince anybody - especially as this is a (mini)game programming test, and with games you can always add one or ten more things.

Its a one-hour hands-on programing lab via screenshare - a pretty straightforward task of consuming some data from an API, processing it a little, and displaying it. We use Ruby for our work, and ask the same of our candidates in the lab. No handwritten syntax, exotic/tricky algorithms, or gotchas. Just a task to see how a candidate approaches a problem, deals with requirements, and implements them in our language of choice.

Happy to go into more detail if anyone is trying to implement something similar in their own hiring processes.

1) It should be a live exercise (at least mostly live). I'm less interested in the end result than how someone got there, and most people chose to narrate their thought process as they work through the exercise.

2) The candidate should be able to work in their own environment. Online assessment tools can be different enough to introduce friction in a timed interview scenario. I'm interested in seeing how fluid a candidate is with the tools they'll use every day.

3) It should have the same expectations as your day-to-day work. Did the editor have a linter configured, and did they use it? Were tests written for the parts of the lab that introduced nontrivial logic? Is there a README? Is it all one git commit, or did they habitually make commits as they worked?

4) The design of the lab should have a decision point - I'm currently using a lab which could be implemented successfully with either a relational database or some kind of cache store. The strongest candidates recognize there are multiple paths and can justify why they chose the one they did.

This works best for reasonably experienced devs who can actually be expected to create something from scratch from a set of requirements on a short time frame. For more junior people, I'd typically give a version of this lab that I completed myself and introduced a couple of bugs into. The task is to fix the bugs.

I really enjoyed this session, but I can see how it could be stressful to others.

So we set up a small project using this framework, and mildly wrecked it. Made some really basic mistakes, made some more subtle mistakes, added a bunch of dead code, wrote good tests for some of the broken stuff, wrote broken tests for some of the good stuff, etc.

Candidates came in, and we asked them to add some simple feature to the app, which we warned them may be broken. We had someone sort-of-pair with them as they work, to watch what they do and keep them on the rails to an extent. We then evaluated whether what the candidate did was sensible: did they run the tests, did they pay attention to the failures, did they find and fix the mistakes, did they get distracted by the dead code, etc. Candidates didn't have to do everything perfectly, but there is plenty of scope to go really wrong. I remember one case where the very first thing the candidate did was delete all the tests. Thanks for coming in, we'll let you know!

The company did quite a lot of rescue missions, where we would pick up a project that another consultancy had started and failed to deliver, and get it live. This interview wasn't just a trick, it was testing everyday skills!

I had a company do this and it went really poorly for me. It was a python app and at the time I wasn't very expert in python. They told me to use my laptop and all I had for python was a vim environment. I knew enough python to do white-boarding problems. But debugging a complex app exercises a whole different set of skills I just hadn't developed in the language. I was also learning vim on the side. I looked like an incompetent amateur.

So there's a language and tooling issue. The bigger problem is that debugging an unfamiliar code base under time pressure is almost as niche as doing algorithm problems under time pressure. I've been on enough company wide on calls that I actually think I'm pretty good at debugging unfamiliar code under time pressure. But if I have someone who understands the code looking over my shoulder, they're usually at least trying to help.

It's still time pressure. It's still a somewhat contrived problem and code base. The interviewer is still comfortably omniscient. And it introduces some more complexity due to unfamiliarity of the language, tooling, code base or problem space.

Ultimately, I'm not sure it's any better than white-boarding.

A lot of these problems are solved with a take home question but then all the problems with take homes rise in their place.

Obviously no one is going to submit an app that doesn't accomplish the primary task. I feel like you get judged more-so on how you decided to get the initial app up and running, which likely isn't something you'll need to do working there.

I would have guessed the same thing before my company started doing homework interviews. But in fact, the ones that don't get to the finish line are sometimes the most interesting ones to review, and they aren't an automatic no-hire. (And we say so explicitly, if a bit more diplomatically, in our instructions.) If someone hands us clean, well-thought-out code and it's clear that they honored our time limit and just didn't quite get there before the clock ran out, we'll still talk to them.

Our test is actually quite simple and everyone that did any kind of development should have done these basic things.

But it tells me a log of the current state of the candidate. Most code samples on internet are not production ready code. Security and error handling are mostly not included. So this is where a candidate can shine... but I'm happy if you can string it together working even if it is by googling.

So you're saying spending 3 and a half hours on the assignment is fine. It's not.

Take-home assignments have little to no value for the candidate, they're only feasible if you are jobless or in college. The last thing I want to do in my spare time is to stay focusing on a bogus problem that may end up discarded.

Talking on-site or on the phone? Is that better spent time

https://www.quora.com/What-is-it-like-having-a-practical-int...

Last year, I did take home test for 5 different companies and all 5 of them invited me to the next round. I yet to have a successful whiteboarding interview.

Just like onsite interviews. Any interview process will require candidates to spend time outside of work. The only way to fix that would be a true, universally recognized certification/diploma that would prove that you can write software, but I don't see that happening any time soon.

And unlike an onsite interview, I don't have to take time off during working hours to do them.

My mental model is that I'm willing to spend N hours in total for the job application process. I'm happy to spend some of those hours working on a take home problem as long as that is instead of and not in addition to spending those hours doing multiple rounds of interviews.

A good middle ground would be to have the company make a donation to the candidate's choice of nonprofit.

How is this enforceable? Even in the case that someone finds out about it, what’s the company going to do? Sue the interviewing company for the rights to a few hours of work?

What are the damages?

As for suing you? Again what are the damages?

It’s almost certain there are no damages assuming you are just working on a take home project not something that they are going to actually use to compete with your employer.

Just because something is in the terms of your contract doesn’t actually mean you really need to worry about a lawsuit.

A company might fire you if they find out, but the chance that they fire you is to small to worry about.

And even if you don’t do take home work for pay, you don’t want your company finding out you are interviewing. Taking a token payment doesn’t change anything practically.

People have contracts with all kinds of stupid things in them. If they aren't practically enforceable, they can safely be ignored. Any company saying they don't pay for take home projects because of employment contracts is just using that as an excuse not to pay.

There are people out their with contracts that say they can't even interview at rival companies, how many companies out there refuse to conduct interviews because of this?

Donating to charity has tax implications for the company too, but they're much simpler.

And for that matter, giving real-but-unpaid work to do as part of an interview may be illegal. So paying for your time may be something companies are doing to protect themselves taxwise, not just for good will.

Now one worries, is this itself a test? Are they judging my desire to donate.

It's kind of like at salary negotiation. You can impress them with how you really are there for your passion not materialism. Or you can get money.

Then I'd be worried... are they watching to see whether I engage in hollow virtue-signalling?

I think this is a key piece why this kind of project is useful, because that's exactly what they'd do when they start in their coding role: jump into an unfamiliar code base and fix a bug, implement something tiny, make things better along the way.

When hiring for a React role, I went to find an open-source React app and simply asked the candidate to mod it a bit. We then discussed the why and how, but also how they liked the app and why.

We only gave this task to the final candidates, and the ones that had React experience were done in very little time, because typing wasn't the issue here, but reading unfamiliar code and understanding the mechanics of the app.

Writing a hello world app from scratch or coding fizz buzz doesn't tell me enough.

I mostly understand why some companies do the x hours routine, and when I've done interviews this way I've historically performed well, but it just felt like it added unnecessary stress. For instance, anyone on the job doesn't have to worry about issues like, "What if I couldn't get the app up and running because I had the wrong version of X installed?" longer than maybe their first day on the job. And if you plan on hiring an engineer for a few years, one day is irrelevant. At best this felt like it favored contractors who were more experienced at jumping into new projects frequently.

Take homes are not very respectful of the candidate's time and, especially if they are not time bound, often leads to a candidate wasting a significant amount of time on these assignments. Plus, there's always the possibility that desperate candidates would cheat and have the assignment done by someone else.

I would only use take homes and HackerRank style quiz for applicants that are from institutions I don't fully trust with quality or applications that are very spammy in nature, i.e from a college/bootcamp that has an abysmal application-to-hire ratio [0].

For candidates from serious institutions I favor a two-step approach: a screening interview with a light coding question and a full whiteboard interview. The screening's purpose is really only to answer questions about the hiring process and check that the candidate can write very simple code by himself when given a trivial problem [1].

Now, a lot of folks do whiteboard interviews wrong. They often expect to get the exact implementation of an algorithm they found in a textbook and for code on the board to compile. This isn't the point of whiteboarding. Doing this only promotes rote memorization. A good whiteboard interview should be a toy problem that can be solved in several different ways by using different strategies or data-structures. The idea is to see how the candidate will break down the problem. Is the candidate able to formulate test cases, write a simple implementation, verify his code and correct the implementation should it fail a test? On the more meta side of things is the candidate able to take feedback and explain why a certain strategy was chosen? Of course it's not representative of real world engineering but it's a good way to peek at someone's ability to debug and reason about programs; these abilities translate well into debugging and design. Especially at the college level, I really can’t make any assumptions on what the candidates know. I’m not judging their knowledge of the standard library of X programming language or the framework-du-jour but their ability to learn it fast.

[0] https://economictimes.indiatimes.com/tech/ites/95-engineers-...

[1] https://blog.codinghorror.com/why-cant-programmers-program/

> I'm actually a little scared to leave my current FAANG gig for that exact reason tbh. I'm fairly certain I wouldn't make it back in the door without more leetcode grinding + repeated loops than I'm willing to do at this point in my career.

>> I'm in a similar position - a tech lead role at a Big N - and have recently been doing some interviews for senior roles at FAANG. The most frustrating thing is knowing I can't really leverage anything I've learned over the past 7 years of my career in the interview, at least at the early stages.

>>> I feel extremely little of what I'm doing in my real, actual, job, helps me in advancing in my career. Unless maybe if I choose to stay at my current company until retirement lol. Otherwise, why bother doing anything more than the bare minimum to get by at work? It would be a far better investment of time and effort to grind leetcode and practice for interviews, instead of going above and beyond to excel at my job. At least until I get into an "endgame company" where I feel it's worth staying long term.

https://news.ycombinator.com/item?id=23848717

Even as we debate the validity of the metrics measured by technical interviews, it's also worth considering the metrics used to measure engineer worth in the industry at large. This existentialist "fear and loathing in FAANG" discussion was quite illuminating, and a little sad.

Makes you question if this is the precise point of it.

Take home project? There goes 4-12 hours per company that gives you one and sometimes more. Companies have no incentive to cut it down or not give it to even marginal candidates so you'll get a lot more of them than full in-person interviews.

Pair programming? That's basically white boarding with a better white board or, in the remote interview world, just regular white boarding. Sure the problem is more real world, in theory, but then you get issues of knowing the same frameworks, etc. So to do well you need to spend a lot of time beforehand studying their specific frameworks and code bases if they ask you to do an issue on their open source project.

It is very easy to completely forget all about graph algorithms in, say, 3 - 5 years during which time you've been a productive, valuable member of a software development team that...you know...actually builds things customers care about.

Being a productive, valuable member of a software development team is teaching you things that will set you up for success in higher level roles. You won’t be taken for those roles on Leetcode interview performance alone.

I think the problem is that for some teams the entire process is some combination of leetcode-style "technical" interviews and then some "culture fit" interviews, without any chance to talk about actual dev work.

You’d be surprised.

I mean, maybe hard or esoteric easy questions but I don't think that's true for basic things like 'find the first duplicate' or '-traverse- this graph'. When people rail on whiteboard coding, I think they mean obscure algorithmic questions and not fundamental data structure questions.

I don't expect someone to be able to do Dijkstra's algorithm (or even spell Dijkstra, did I do it right?) but everyone should be comfortable enough with a bit of help doing BFS or working with linked lists, hashmaps, no? Am I crazy here?

It's not really that difficult to make quotidian software engineering tasks into interview questions. I've at least seen it done successfully before.

More anecdotes: a nontrivial number of the "algorithmic trick" interviews I've done were questions I knew the answer to beforehand – and often the interviewer didn't care when I told them ("I just want to see how you think" or "Just show me how you'd code it" or sometimes even "OK, just tell me the solution and we'll move on to the next part"). What happens to the people who don't have the same background? Are they really less competent software engineers because their algs class didn't cover making a queue from two stacks?

"Find the first duplicate": In my actual SW roles, this was trivial: Use a hash/map/dictionary/set. However, this isn't a typical Leetcode question - often it will be with additional constraints (cannot use a set, need O(1) space, etc). I don't doubt there are domains where this is important, but I've never had to do this in my professional work.

"Traverse this graph": Probably had to do it a few times. Less than once a year.

BFS: Never for my job.

Linked lists: Never for my job.

Yes, it is good to know what a linked list is and know if your library's data structure is using one, but again: That's not your typical Leetcode question. You'll be expected to write a linked list to solve some problem.

I do get what you're saying: Stuff like linked lists and BFS are fundamental, even if most SW jobs almost never require them. It would be a red flag if a candidate didn't know the basics about them. It's another thing to ask them to code it on the fly.

Going back to my electrical engineering experience: We have to take and use a ton of calculus to get the degree - much more than a CS grad will take algorithms. Yet it's a common trope that most electrical engineers will never use calculus on the job (literally not hard to find one who had a whole career in EE without using calculus). If you're doing research or are at the forefront of a discipline - sure, you'll need calculus. Or if you're doing electromagnetics, etc. The theory is built on calculus, but you don't need to know it to apply the results of the theory.

Would it be a red flag if a candidate clearly didn't understand the basics of differentiation (i.e. rate of change) and integrals (i.e. summing quantities)? Perhaps.

But should I ask someone to do a tricky integral for an EE role that will not require the person to do any calculus? I would be criticized by my peers, and if I kept doing that I would be excluded from interviewing. In fact, even if I ask them to do non-tricky integrals I would be criticized.

Asking people to figure out a trick to solve a linked list problem is equivalent to asking EE candidates to do tricky integrals or differential equations.

Eventually I changed the strategy and offered candidates the choice of the take-home problem or an on-site whiteboard style interview where they solved a subset of the problem in pair-programming fashion with us. Zero people chose the on-site interview. Everyone chose the take-home problem.

No one complained any more after we reframed the take-home problem as the candidate's choice rather than the company's.

Ironically, you passed on the smartest applicant of the bunch.

> reframed

Aren't meaningless shenanigans generally associated with meaningless things?

Do you have any evidence that either strategy really matters? Does it boil down to, "We choose an interviewing strategy that selects for people who look like programmers."

Are you meaning they wanted your company to pay for the time used to solve the take-home problem?

If so, isn't it normally done that way in order to prevent companies potentially exploiting free labour?

Do you offer candidates a remote pair-programming interview?

E.g. screen-sharing session over Skype/Zoom?

If i'm really serious about joining a company and the many months/years to come with them, I can probably spare a day to show it.

It's always surprising to read people complaining about spending even 8 hours on a take-home problem in the comfort of their home, with no one looking over their shoulder, finished at their leisure. Meanwhile, anyone serious about applying to the same company wouldn't hesitate to take PTO and leave work to go on-site for a similar amount of time for an interview.

And then they didn't even ask me any questions about my take-home solution--it was just ignored. It felt like a pure waste. I'd much rather do whiteboarding.

The homework is no cost for the employer and you are not sure how serious about you they are when you do it (if it is in the start of the process) so I wouldn't bother if I had an alternatives to go for.

I personally when I interview candidates prefer to get to know them and how they think before I even try to understand their ability to problem solve. Knowing the language or algorithms are all easy to learn things. Knowing how to naturally be curious and solve problems is a far more difficult skill to learn and demonstrate, but it is easy to evaluate indirectly.

That's what leetcode used to test and then people optimized for it and now it tests memorization. At scale things work differently than in isolated cases. I've also found that most problem solving interviews really test how well I've seen the problem before and how well I can lie about seeing it before. Interview enough candidates and the good problem solvers will look dumb compared to the ones who simply saw it before and can act like they didn't.

We place the candidate in a room with a large whiteboard for 15 minutes alone with the lights off. I then walk in with a 6 piece dresser [1] from IKEA (obviously keeping the lights out). I explain to the prospect that this particular dresser is missing one piece - BUT we need the dresser assembled then disassembled and placed back into the box [without anyone seeing]. More than HALF the candidates become violent and demand to see my github account - some even suggest I have a mental issue. I am often able to calm the candidate down - this way we can get into the REAL debate: KOPPANG vs MALMs. Any candidate caught assembling the IKEA dresser is quickly dismissed out of spite. This technique is from Boz at facebook - it is how all* senior principle engineers are screened at FB and the reason everyone codes PHP (but swears they do not).

HR has complained that my techinques are "ridiculous" or borderline illegal - but so far my successful candidates perform, tend to have a solid grasp of typescript and quickly move into senior management.

https://www.ikea.com/us/en/p/koppang-6-drawer-dresser-black-...

People want hiring processes to be somewhat consistent, and in that world it's really kind of a non-starter to replace tech screens with GitHub profile reviews.

for that matter, GitHub is useless if the projects you contribute to are on BitBucket or GitLab. that contributions graph has a lot of assumptions baked in.

Maybe the solution is for companies to hire more thorough managers who are capable of vetting a github profile to see if the candidates skills are relevant to the position, not turn the interview process into a performance.

So anything on github that I might want you to look at? I’d have to be on a three year replacement cycle, permanently, which is a hell of a lot more work than you might estimate that a reasonable github presence would entail.

As an introverted and generally very socially anxious developer, this sounds like a dream compared to a standard interview, which has always felt to me more like an interrogation and generally very aggressive and hostile. I'm confident in my skills and I can communicate well in other situations, but the typical interview environment is cripplingly stressful to me. I would love to just get through a talkative lunch instead.

Going through my GitHub account however isn't good. It's full of quick one shot things, experiments and so on, very little production ready code and not representative of anything.

My current employer doesn't have explicit rules that I'm aware of, but early on, I asked our HR person if I could work on open source outside work and they said something along the lines of "what's open source?" Which, I mean, on the one hand what do you expect from HR, but on the other hand, the entire agency is concerned with operating, maintaining, and developing a software system.

If it's not on employer-owned equipment during core working hours, then I believe it's pretty difficult for an employer to enforce that kind of thing. I don't know how important to you working at that charity was or participating in open-source now is to you, but if you're willing to take a harder line of telling rather than asking, you might find that when push comes to shove, there might not be much they can do without drastically escalating the situation (not always an organization's favorite thing to do, even if they can technically win, which they might not be able to).

Obvious disclaimer about being a non-expert on this sort of thing, but really just wanted to point out that the first answer in this situation may not actually be the final one.

Also, running side businesses was not allowed.

Of course, the normal thing to do was just ignore the rules, because most likely nobody would notice or care. I explicitly asked for permission because I wanted to see what would happen.

And its not really the "employment contract" in Anglo Saxon legal systems its pretty much assumed.

This might require engineers unionizing, but it would be a boon to the field. Patterns buried in closed source vaults would be disseminated and innovation would surge. It might usher in a golden age of software if everyone were able to freely share their intellectual ideas with their peers and not have to worry about risking their health insurance and home in the process.

That kind of interview mitigates the amount of trivia and preparation-gaming while also ensuring that the applicant can solve problems as a team and is a competent programmer. It is cheatable in that one can memorize implementations of some features, though it seems somewhat difficult and not more cheatable than just memorizing a bunch of CS trivia.

I think part of the problem is not everyone is good at giving interviews. But most software engineers can talk about their work once you get them going.

FWIW, I think white boarding interviews are pretty fun.

Ask questions about the code:

- why did you write this in this way

- explain how this works

- if you needed to add X into this, how would you do it

- are there other ways to achieve what you did here, what are the pros/cons of this?

- what would you do differently if you were to re-write this?

- are there any bugs or common issues, how do you go about diagnosing and debugging them? Could you write the code differently to be more resiliant to those issues?

Anyone who wrote that code (assuming it was relatively recently) is going to be able to talk about it sensibly, and be able to dig into the code and show how certain things are done.

It was very obviously there as bait for people like you and it was removed as soon as they took a new job. It was never used in production in its "released" form.

I had no problem with the person's abilities to code, but for reasons like this I probably wouldn't work with them again, or hire them as a eng lead or higher.

I put the anecdote up as a real-world example of one way the GGP's github reference can be faked.

You would have to dig really hard into their knowledge of the system to catch them, which you wouldn't have time to do.

For instance, I believe the issues would show up in a benchmark.

I'd bet they could also pass a whiteboard, pair-programming or take-home test too.

That they are willing to lie about it and pass off other's work as their own isn't something that you're going to identify in your typical code interview. Using interrogation tactics to try and weed out the frauds is going to be counter-productive, by driving off all the good candidates.

But I'm not sure, I've never interviewed this person.

Sprinkle in some bugs. Start with easy spelling errors and go all the way to concurrency problems.

Ask them to add a few simple features and fix any bugs they find. Bonus points if they find bugs you didn't add intentionally. Good unit tests are also a bonus.

Give them a couple days or something. Don't count off if they don't get everything done. It's better to do a few things correctly that fix everything with hacks.

Invite them to email questions/suggestions to the hiring team. This person might be on your team so it's nice to find out early if they have good ideas or they hound you with stack overflow questions all day.

I've never had this kind of interview but it sounds nice to me. It's more realistic to real daily work. You can cheat by having someone else do the work but that's always a possibility.

Whiteboardimg a solution to n=np or building full applications is crazy and not realistic. Most interviews do that yet most work is fixing bugs and adding mundane features. Why can't interviews focus on the actual skills needed?

What’s your threat model here? People making up elaborate lies about code they’ve written?

Likely people making up elaborate lies about code they didn't write. It's trivial to push code of which you are not the author.

Why would I leave a stable well paying job for a job that basically says they don't trust me enough to not fire me in two weeks?

Manipulating one or more git repos to claim it as yours then ensuring your lie is not uncovered by googling would be technically challenging enough that those who could do probably don't need to.

Without looking at the commit history, given enough time this is a challenge. I often don't recognize code I wrote myself 6 months ago.

It obviously depends on your definition of "better". And it's yet to be seen whether whiteboarding is "better" than pulling candidates out of a hat. You'll note that lots of companies that do strict whiteboard-style interviews are also extremely comfortable firing employees. Would you be "better" just hiring anyone whose resume seemed applicable, that could talk like a normal person, and then firing them if it didn't work out?

Do you have any statistically significant data to back this claim?

That whiteboard interviews are actually testing how you perform under stress rather than how good you are at your job.

Additionally, in my opinion, it measures how good the interviewer is. If you get a bad interviewer or even just one in a bad mood, just pack your bags because you aren't getting it.

Whiteboards are incredibly arbitrary in my opinion, if taken alone. If it is part of a complete assessment, ehhhh.

Interesting study and really good discussion on HN at the time, but the opposite of "[Whiteboard interviews] often give a very good and reasonable signal that often has a high corollary to work performance" would be something like "People who do well at whiteboard interviews tend to be worse performers at work." I don't think that's what the study said at all; rather that people do worse at whiteboard problems when they're being watched vs. not being watched.

Not. All. Anxiety. Is. The. Same.

Ask a firefighter to give a speech in front of 1000 people. This is an entirely different kind of anxiety than running into a burning building. Ironically, it may be the former rather than the latter that makes the firefighter sweat.

Writing code is generally not a performance art. We're not stage performers. Interviews are an unusual situation that aren't anything like our daily jobs. Stress on the job is not the same as interview stress. Work emergencies are not the same as interviews.

We need to take into account human psychology. These factors are all well known, but the tech industry seems to act in deliberate ignorance of humanity.

One you are getting paid no matter what, the other you aren't and are 'performing' in the hopes of getting paid.

Assuming the study’s results are valid, that’s really all the justification we need to get rid of these tests.

if properly trained, interviewers ask questions that build without throwing the kitchen sink at you.

whiteboard interviews (when done holistically) assess for signal, not binary correctness

- do you have a solid framework to build a solution?

- are you considering multiple approaches / data structures / complexities

- referencing similar problems, vocabulary, situations to illustrate breadth of knowledge

sure, the stress component isn't ideal, but there are multiple relevant capabilities being assessed in a very short amount of time...

in other words it's a much better test of tacit knowledge than alternatives

if done right, whiteboarding should be almost the same as pair programming -- an iterative dialogue

WB interviews should be about problem solving not regurgitation of rote learned algorithms

I won't ever fault a candidate for failing to close a paren or being unsure of whether the size of a container is .length() or .size(), but if virtually every line a candidate writes is of questionable value, I'd say that means something.

One of my earlier jobs dealing with code involved a very brief interview (no whiteboard involved) followed by a 6 month contract offer. The deal was that I basically had 6 months to prove myself, and if everyone felt like it was working out (myself included) this would be bumped into full-time with benefits. The crazy thing is, I was actually the one who decided I did not want to continue after the 6 months was out while they wanted me to stay very badly. I think this can be a really useful checkpoint tool for both sides.

Anyone can misrepresent themselves in an interview and dupe a room full of people for a day or 2. No one can keep up an act for 3-6 months continuously when working with complex code tasks. You will get found out before the term is up, assuming everyone is using this process as an evaluation tool and following up on it regularly.

It also puts a candidate at a disadvantage, because looking for a job is heavily one-sided burden.

On a personal level, as both (occasionally) a candidate and a senior hiring manager, I always wonder about white boarding - i believe it works when the interviewer is looking for your thought process, and not code specifically. For example, something as simple as saying “I don’t expect this code to compile” at beginning of interview immediately puts candidate at ease and let’s you understand their logic and not the ability to recall specific function signatures on the fly.

It seems like a reasonable compromise that doesn't leave the job seeker without health insurance etc.