Google did the "how many gas stations in the us" interviews for years before finally realizing that doing well on questions like this had no correlation to success on the job. Now, in reaction to that failure, everyone is doing the LeetCode thing instead, and presumably will eventually realize that this doesn't correlate either (especially irrelevant in this CoPilot era when ability to memorize and code up an algorithm is becoming about as relevant as ability to drive a stick-shift car).
There's really no substitute for interviewing based on candidate's experience.
Apparently the new job market trend is applicants firing off dozens/hundreds of AI generated applications, which are then screened by an AI on the other end.
Haha I’m not sure I agree about how useful CoPilot is in practice (there are certain tasks where it shines but all of the LLMs print invalid code routinely).
With that said you made me laugh out loud because I have a friend who knew two people trying to LLM some minor contract and it just went on forever. I definitely think two LLMs talking to each other with two people copy pasting and not realizing it is worth like, a Rick and Morty episode or something.
Yeah, I've even seen people in top ML jobs glowingly describe the main value of CoPilot as (just) smart autocomplete, but copying/regurgitation of algorithms ought to be something that even today's LLMs should be capable of.
Of course there's always Google search too if you are just looking for an algorithm, but LLMs help in the discovery process since you can describe what you need without knowing the name of it.
Using LLMs to find reasonable subset of algorithms for a problem sounds like a valid use case. Could even get naive implementation and lead to comparing them to pick right one.
If they take the LLMs advice were they astute or just lucky? You'd have to ask the candidate to go into details to get any signal, and if the algorithm is hard enough for them to have to consult an LLM, they might not be able to do that.
I was referring to LLM use on the job (or hobby), not as any part of interviewing process.
The reality is that most programmers can go an entire career without being algorithmically challenged, and on the rare occasion you need to do/optimize something and don't know what the best options are, then you can either just ask a colleague, or ask online, or Google, or nowadays discuss it with an AI!
Why do you think they haven't adjusted again? Is it possible it is actually (loosely) correlated to job performance?
I interviewed at Google and all the questions were practical, challenging, and not found on leetcode.
I've seen all kinds of interviews in my twenty years of experience, and while all types can be done poorly, (including DS&A ones), having some live coding is one of the best signals you can get in a short amount of time.
I've been blown away by a candidate while they're talking about their experience, but then asking them to code something small, they utterly bomb.
Conversely, I've seen shy, humble candidates struggle to articulate their skills, and then crush harder and harder coding challenges.
> I've been blown away by a candidate while they're talking about their experience, but then asking them to code something small, they utterly bomb.
You're making the opposite point you think you are. How do you know that you observed an inability to code as opposed to interview performance anxiety?
I've been coding for 25 years, but have bombed my share of very simple white-boarding exercises because I panic and my brain absolutely stops working. If it's experience-based or even take home, I generally knock it out of the park.
White-board coding works for FAANG because they have 100,000 applicants at any given time and it doesn't matter if it has an exceptionally high false negative rate. It does not work so well for shops that are struggling to find candidates.
>You're making the opposite point you think you are. How do you know that you observed an inability to code as opposed to interview performance anxiety?
I've done lots of interviews, and I know that if someone has seen a question recently, or if someone is rusty in interviews, or one of a million things can affect someone's live coding ability.
So I'm not looking for speed, I don't give more 'points' to someone who is calm and collected vs. someone who takes twice as long cause they're nervous. They both "pass" in my book.
I also don't particularly care if someone starts intermixing java and C++ syntax or forgets the string library and says len(var) instead of var.len() for example.
I'm even fine with giving hints. But there are literally people, with hints, who cannot write a basic recursive function or understand to check for null references. That isn't being nervous. That's just...being bad at the craft.
It absolutely does not always happen. I was asked to do interviews twice with one employer. Both times, I requested the interviewee's resume. Both times I was refused because they didn't want me to "bias" my interview.
It’s not generally the responsibility of the person doing the technical screen to also obtain signal on past experience. At a company with recruiting process these fall on different people specialized in the specific interview types. I’ve done an average of 2-3 interviews a week over several employers for the last 12 years, so over a thousand interviews. I’ve also defined much tech interview loop for my current employer and usually interview from junior IC up to manager of manager roles.
I’ve never asked for candidate resumes because I don’t want to bias myself.
I’m not saying it will always, always happen, it’s just, it overwhelmingly will.
If you don't have a resume, do you still ask about candidates experience?
Many people can talk a good game and come across well but still be useless (e.g. hard fail on the simplest of white board tests).
I haven't done interviewing (on company's side of table) for a long time, but my main tack was always to go over projects on resume, focusing on last 5 years, and drill down to see if they could fluently talk about them, draw architecture diagrams, justify choices, etc, etc.
I you don't ask about past experience (= same as having resume), then how do you even know what the person's experience consists of? Useless cog, or tech lead? Highly agentic or passive, etc?
Maybe we were talking past each other a bit — for the specific technical portion of the interview where you are assessing coding ability (the thing you can leetcode to practice for), the resume doesn’t really add much value and can serve to bias the interviewer.
There is usually an experience and goals interview where a manager interviews the candidate to see if they would make a good fit for the team. That definitely involves a resume and a discussion of past work experience with the objective of confirm if they’re a cog or a tech lead :)
It can be important to split them up especially if you’re relying on more junior technical interviewers to validate coding skill. They won’t always know what they’re looking for yet along this axis.
There's really no substitute for interviewing based on candidate's experience.
Apparently the new job market trend is applicants firing off dozens/hundreds of AI generated applications, which are then screened by an AI on the other end.