The study participants were split into two groups. One group solved the problem alone, in silence. The other group was instructed to narrate their thought process out loud to an interviewer who stood over their shoulder (literally, see the photo on page 3) while they solved it.
The flaw in this study is that the two groups weren't really assigned the same task. Solving a problem isn't the same as verbally narrating your solution to a problem. The authors attributed the differences not to the extra work narration work assigned to the second group, but to anxiety levels.
The participants even explained that their difficulty came from having to talk while solving:
> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”.
There are other concerns with the study format such as many of a significant number of participants simply giving up well before the clock ran out, in both the private and public interview groups. They were given 30 minutes to solve 3 problems, but some of the participants (including in the private group) were giving up around the 10 minute mark without solving anything at all. This suggests a very high variance of the underlying abilities of their candidates, which necessitates a much larger sample size to draw conclusions.
It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach.
The last interview sequence I needed to do they put me in front of an online IDE and said "go code." They also said I could use my OWN IDE if I had it set up. One even said "you can finish it later (post-interview) if you want."
The candidates giving up: Some nontrivial percentage of CS students are just ... not capable developers. That's why we use programming tests to begin with.
Programming tests may be the worst possible way to assess candidates, but they also seem to be the only way to assess candidates that doesn't involve paying them for a trial period, which really only works for companies large enough to be able to waste a lot of money. "Democracy is the worst form of government, except for all the others."
Having candidates talk through the solution to a non-trivial programming problem is a critical tool for me. I need to find developers that can not only code, but work as a part of a team - something that requires significant communications abilities.
I can't afford to hire developers that can't effectively communicate complex ideas to other developers - something they're often required to do on-the-fly.
These are two different modes of operation. I can code when in abstract problem-solving mode, and then later communicate that complex idea using my verbal skills. I can't do both at the same time.
It's like asking someone to give an extemporaneous talk on some subject, while translating each sentence into another language as they go. Even native bilingual speakers would have quite some trouble with this. They can generate content, they can speak fluently, and they can translate on the fly, but having to do all three "live" involves too much context switching. Maybe doable for simple or rehearsed topics, but most coding interview questions are non-trivial.
I don't think this is universally true, and to some extent I think it's a practiced skill.
I used to really struggle to problem-solve and communicate at the same time, but then did two years of improv comedy outside of work and suddenly don't have that issue anymore. I think part of it for me was shortening the gap between my thoughts and my communication (to zero) which stops the need to context switch, but introduces a need to think in a way which is clear and structured (because you are going to be verbalising the voice in your brain, rather than thinking and then translating those thoughts to words as a secondary step).
> I don't think this is universally true, and to some extent I think it's a practiced skill.
But is it a skill that's even necessary and therefore developed while working as a software engineer?
In almost all cases that i can think of:
- real time communication might be important for discussing things before and after development, things such as architecture, concerns to pay attention to, or even choices made during development
- real time communication almost never plays a part during the actual development, unless doing pair programming live from 0 or even mob programming (absolutely unheard of in my current company)
- most of the time, what happens instead is dropping a detailed question in Slack and the other person responding to it when they have a minute, only escalating to calls if that's not helpful, which almost always only gives rough outlines of things to try next, not changesets to the code
- actually, most of the communication regarding code happens in merge requests asynchronously, which is better than synchronous communication which wouldn't be explicitly recorded and all of the arguments/outcomes of which would disappear
Thus, narrating the problem solving process live when you don't even have any idea how to solve the problem, haven't done research about that particular problem, have no idea what others in the industry have tried and how that worked out, is almost completely useless in my eyes. During this process, when you haven't done this discovery, about 80% of what you'd think of would be useless and would be tossed aside when you'd figure out what you actually need.
You can only feel confident in your ability to narrate these solutions when you know what they are or where to start, e.g. after a few hundreds of hours of grinding LeetCode, but at that point it's essentially following a flowchart of algorithms and in my eyes is basically "pretending" to be clever, even if there is something to be said about the ability to memorize solutions to problems (or even groups of solutions) and apply the correct one.
Perhaps it's a bit better in my eyes when you'll be applying these algorithms daily in your work, but so far it's about 90% CRUD development, struggling with the specifics of technologies and doing requirements engineering with clients who don't always know what they want or whether that what they want is technically feasible.
Disclaimer: there are probably cultural aspects to it as well, like how here in North/East Europe we don't really do much small talk with strangers. Or maybe it's just corporate culture, or the people working within the company happening to be more introverted and thus the idea of narrating things real time simply being uncomfortable to them.
I think it's probably not a skill required for most software engineering, but it's probably useful if you have a more senior engineering position where part of your role is helping and coaching less experienced people.
It's also useful to 'work a problem' with people sometimes which requires this sort of live communication, however the extent to which teams do this will vary by team. In my experience this is more common in teams that are physically located together in the same room.
Also it's not inherently clear to me that "not used in the job" means it's also bad from an assessment perspective - take defensive driving certifications for example, which require drivers to 'verbalise' what they are noticing, doing, and their decisions while they drive. Do you verbalise when you are driving in reality? No - but it is an established technique for assessment and for learning.
But each to their own. Different strokes for different folks, clearly this works for some people and not others.
If you can code, then explain, but not in reverse - that indicates you don't have a path to solution in your mind without some trial and error. Some levels of engineering require you to have one, at least for certain tasks.
Right, so the first step in this process is to think through the problem until you have a path to a solution in your mind, before you start coding and explaining. But the job interview process we're talking about explicitly asks you to start talking aloud about your thought process (and be judged on it!) with essentially zero time to think it through first.
I code as part of the process of thinking things through. Seeing how bits will fit together. It's like how an author writes to organize their thoughts, instead of the other way around. How ridiculous would it be to interview an author based on their ability to write a short article on a whiteboard in real-time, while talking through their writing process!
For me it has been the other way around: When I give a programming problem for a candidate to solve, I ask them to focus for the 30 minutes in the problem and solve it.
I myself found it asinine when I was asked to talk while writing code... I just cannot focus in the problem at hand while coding. Maybe I am stupid or low IQ, but I need to concentrate in the problem and put 100% of my attention on it.
From interviewing several hundred people so far, I've seen that most of them breathe a sigh of relief once I told them that rule (along with the rule of being able to search for whatever they want as part of the challenge).
The image of a developer constantly speaking while coding on their day to day brings me a chcukle.
If you want a comparable experience, assign a difficult task to one of your developers, then tell them: "I am going to watch you solve this problem. Talk me through it as you do so. I'll judge the result, if its not good enough I will have to fire you on the spot. Also this is timed. Proceed."
>>Having candidates talk through the solution to a non-trivial programming problem is a critical tool for me. I need to find developers that can not only code, but work as a part of a team
So at work, every time your programmers are assigned a JIRA ticket they speak in loud voices every line of code they are writing/deleting/editing???
Sounds like a horrible way of managing a team.
>>I can't afford to hire developers that can't effectively communicate complex ideas to other developers - something they're often required to do on-the-fly.
This goes beyond I imagined. Once the JIRA is assigned they literally scream at each other what they are thinking?
I have had situations where people complain about noisy clicky keyboards, let alone people loudly repeating their ideas on other peoples faces.
This is an obviously ridiculous take on a relatively clear idea: building complex systems using multiple development teams requires effective coordination, which implies not only development skill, but communication of technical plans.
If you're on the team assigned to develop the UI for the turbine monitoring of a hydroelectric power generating station, at some point you're going to need to be in a room with a bunch of other developers and engineers, and start working through some technical issues or begin planning how you're going to approach the problem. That will involve working through code while getting input, which will obviously involve speaking to the other humans in the room.
I know there's a tendency on HN to think that all developers are just like you, but a lot of developers aren't working on simple nonsense like the next to-do app or note-taking tool. Many developers are working on complex, real-time systems that require the coordination of dozens of teams of specialists over years. If you can't manage to communicate your approach to solving a particular problem to a room full of engineers and other developers, you're just not going to do well in that kind of environment.
>>building complex systems using multiple development teams requires effective coordination, which implies not only development skill, but communication of technical plans.
That's not what is being discussed here. As a matter of fact anything as complicated as what you describe here, can't be explained in one cycle. It will require several takes to arrive at something good. And it definitely doesn't involve repeating their deepest thoughts at their colleagues faces.
If anything your interview method feels more like an audition for an acting role than problem solving.
Funny you should describe it like that, but ... what you've just described is almost indistinguishable from pair or mob programming.
Where one person needs to describe clearly what code needs to be written so the other can type it. Actually pair programming requires more effort in that regard, since describing what you're doing is easier than describing exactly how to write it.
I don't do 100% pair programming gigs, but I know they exist, and I've used pair programming in short bursts as a tool for knowledge transfer.
I worked in teams whole my life. Having to explain while I am solving something non-trivial is rare to non existent situation. If just don't happen.
Having to explain something non-trivial happens. But literally always there are hours between solution and explaining it. In pretty much any real situation, I have quite a lot of time to think about how to explain.
Probationary hiring also doesn't work that well for many candidates. Despite at-will employment, there's an understanding about W2 white collar employment that gives new hires the confidence to buy and sell houses, move across the country or around the world, terminate talks with other prospective employers and pause interviewing, etc. This is a cultural thing that could change over time, but it would be a significant change.
I actually did end up hiring a W2 employee who didn't work out. He moved cross country for the job too.
Let him go after only a couple of months--but that was several months of pay that produced literally no positive results. And I was running a small development house at that point; dumping two months of salary down the drain was not a good thing for the bottom line.
So yeah, totally agree. Paying everyone who "seems good on paper but who just can't pass the programming test" really doesn't work, despite wishful thinking to the contrary.
This is a great point about CS grads. Maybe in the past, you could just take a degree at face value: "oh you graduated from Stanford with an A average, you must know how to code, here's a job". But nobody trusts any university to produce capable developers, so you have to test everyone with crazy interviews. So then what's the point of going to a top CS school and why can they keep charging so much tuition?
Some schools have great signal to noise ratio, and you can be pretty confident when hiring. It's not just reputation. Some school (Poly in Montreal is a good example - but other do it as well) requires all student to complete a paid internship in their field of study to graduate. So you know if you hire someone from there that they are able to code professionally.
I doubt there exists a 4.0 gpa standford cs grad that is incapable of coding. It is more than just branding. Standford is harder and you learn more skills that a run of the mill state school.
MIT moreso, no one is skating under the radar at MIT
A past CEO I worked with was a CS grad from MIT. He would laugh about the fact that "he could code, but no one would want to use the code he wrote."
I've also heard another anecdote about an MIT grad who had excellent grades but who somehow couldn't get any work done on a relatively straightforward project.
So yeah, even MIT graduates CS students who aren't the best at code. CS being "Computer Science" means they need to understand the theory, but even an MIT grad isn't guaranteed to be capable and talented in writing code in practice.
If there were a "software engineering" major (or certification?) which, among other things, required the student to write nontrivial code, and it was judged by experts, including using subjective design criteria, then maybe we could start using that degree or certification to prove that, at least at the time of the tests, that person could code. But the devil is in the details even then...
So would you be comfortable hiring a 4.0 grad from Stanford/MIT without doing a coding interview? Just based on their transcript. Does anyone do this anymore?
> It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach.
That idea definitely makes sense to me. And I think some devs do definitely underperform in interviews.
Though I'm not sure if there's great alternatives. As an interviewer I feel that seeing someone work through a problem and debug issues gives useful information. I don't know if there's a better, less anxiety-producing way of testing those things.
I do think take homes can be really useful, but those have their own problems. Maybe leaving someone alone onsite to complete the problem would be an option. But I think leaving the discussion to the end would give less info about the candidate's communication skills.
Just give them an initial five minutes or so to digest the problem themselves before asking them to communicate. Or even a few couple-minute long breaks throughout the interview where they're not under the microscope. Or some other way to lower artificial pressures.
This unwillingness to experiment with the existing interview format, the intransigence to consider any alternatives, is simply maddening.
It strikes me as odd that interviews aren't collaborative. I mean, the really valuable information that interviewers want is whether or not the candidate is going to work well with other people there, right? So interviewers could work with candidates to solve some problem (maybe problems structured so that it's naturally convenient to divide them into two parts where each contributes). An external service could provide problems and software for shared solving to prevent interviewers from knowing the problems in advance (which would revert to them "testing"). And the shared problem-solving could be recorded so that interviewers could review it afterwards to decide more clearly how it went (maybe they felt like the other person did a lot during the shared problem-solving, but when watching it afterwards, they realize that the interviewer carried the weight of solving things).
I think this kind of context change would be tremendously empowering for candidates such as myself who get very anxious about situations in which they feel people are "testing" them but who actually love working with people. It would also be nice as a way to see what people are like to work with, from both the perspectives of the candidates and the interviewers.
Very excellent point and good ideas for a solution. Funnily enough, I've recently had the same thoughts, been wondering what it would be like if interviewers were asked to work with interviewees on problems they were equally as unfamiliar with, so they are on equal footing.
That would be a flaw in the sense that anxiety may not be the right word, but it does capture the parallel with white board interviews and underscore the problem that you may be turning away people who can perform the job well but not code while explaining.
Also the effect basically disappears if you take only male candidates, so the study is a bit suspect. Men and women typically aren't that different in studies.
I have read a lot of psychological studies, the effect size difference between men and women in the same study is typically pretty small. But in this one the effect is close to 0 for men and 100% for women, that is really suspicious. I wouldn't trust results like these unless they were replicated with more robust methods.
> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”.
This is why I built myself a tool to practice talking and narrating my thoughts! https://enumerable.co
> The flaw in this study is that the two groups weren't really assigned the same task.
The real flaw is that the title doesn't follow from the research. "Tech sector job interviews assess anxiety, not software skills." That's saying that you're really measuring level of anxiety, not skill. So skill is irrelevant and you're just measuring how anxious people are.
But the study didn't show that at all. It just showed that on average, people perform better when they're not anxious. Not exactly a surprising result!
Of course people are going to be more anxious in an interview. But if it affects everyone equally who cares? Just make your questions easier.
maybe you should get in touch with the professor and tell him? curious what the answer will be.
i think it’s reasonable to think that this study is relevant and that there is truth in it.
there is also the issue that every interviewer i encountered assumes lying and incompetence and will conduct the interview in such a way that the result is what they want to see - the candidate failing to write the code within a few minutes. they are not capable of letting a candidate do the interview without biasing the interview and results.
this also comes from the automatic assumption that because they work in a FAANG they are a fucking genius and are the only ones who can write software. i can see this happening as they are so brainwashed and lost all touch with reality and how the rest of the industry works.
meanwhile there are some marketing execs at that FAANG with a compensation package far above what they will ever get paid.
The study participants were split into two groups. One group solved the problem alone, in silence. The other group was instructed to narrate their thought process out loud to an interviewer who stood over their shoulder (literally, see the photo on page 3) while they solved it.
The flaw in this study is that the two groups weren't really assigned the same task. Solving a problem isn't the same as verbally narrating your solution to a problem. The authors attributed the differences not to the extra work narration work assigned to the second group, but to anxiety levels.
The participants even explained that their difficulty came from having to talk while solving:
> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”.
There are other concerns with the study format such as many of a significant number of participants simply giving up well before the clock ran out, in both the private and public interview groups. They were given 30 minutes to solve 3 problems, but some of the participants (including in the private group) were giving up around the 10 minute mark without solving anything at all. This suggests a very high variance of the underlying abilities of their candidates, which necessitates a much larger sample size to draw conclusions.