From that particular post, this line stuck out to me:
Only 3 candidates were able to pass this code filter. The people that did pass either did well enough to pass or extremely excellent. The people that failed, maybe 19 of 22 people, all failed epically.
In practice I highly doubt that the population of candidates is actually this bimodal. It might lean heavily below the hiring bar, but I have trouble believing that it's bimodal.
My own personal experience has been that it looks a lot like a normal distribution where the hiring bar is just two standard deviations above the mean (which would imply that you give offers to about 1 out of 25 people who apply, which sounds approximately in the right ballpark).
As a consequence, if your question is producing bimodal results, I can't help but wonder whether it's a good interview question, because knowledge tests exhibit much of the same type of distribution of outcomes.
In my experience, if you give the candidate only one hour, access to documentation only helps with recalling past experience, because one hour is too much time pressure to actually calm down, pick through, and digest documentation on something you haven't seen before. If you knew the DOM functions existed and just needed their names, it'll work, but if you're unfamiliar with what the DOM already offers, even if you could learn it fine on the job, you won't learn it fine under time pressure. I recently adjusted one of my interview questions specifically to reduce the need for looking up documentation because I noticed exactly this phenomenon.
It surprised me just as much. I cannot speak to the capabilities or thoughts of any given candidate. Ultimately the test was a test of using a single standard API that is based upon a standard model. If it were just a matter of instructions or syntax I suspect a candidate would gotten over that by either asking the right questions, trial and error, using a reference of whatever. I think what really destroyed people was a complete inability to perceive the page as a series of nodes all interconnected with various relationships.
> In my experience, if you give the candidate only one hour, access to documentation only helps with recalling past experience
That is only partially true in this case, because the candidates knew coming in they would have a code filter accessing a page using vanilla JavaScript. I cannot remember if I specifically mentioned refreshing on the DOM methods or not. I might have. This was 8 years ago. But they had at least a day prior notice to catch up on how to interact with a web page using JavaScript using only their code. The time pressure would have applied to the specific tasks asked of the candidates.
I think what really destroyed people was a complete inability to perceive the page as a series of nodes all interconnected with various relationships.
Fair enough. :) I've experienced a similar issue with interview questions where they often floundered on the question because they hadn't ever spent time digging under the surface. For example, I notice that a lot of (typically junior) developers seem to only think of JSON as a representation for hierarchical objects but never really spent time thinking about the typed-ness of its keys/values or the limits of what JSON can actually represent. An interview is a bad time to be confronted with that realization.
I cannot remember if I specifically mentioned refreshing on the DOM methods or not.
Well, it was a long time ago, so I'll give you the benefit of the doubt and just talk about what I've experienced. I go back and forth about how specific I should be about the type of questions I might ask. On the one hand, it seems kind of foundational that someone writing web frontends is aware of the DOM and how it works, even if they never directly interact with it. On the other hand, I think questions need to control as much as possible for the "preparation" factor because some people simply interview better or have more time to study.
It doesn't seem too crazy to me today that a competent FE dev these days might have spent their whole career in React, and therefore they don't really have a strong grasp on the DOM. If you had asked when I first started doing web FE development, that would have been crazy! But frameworks got a lot better and a lot more comprehensive since then.
I recently had an interesting experience asking a question involving dealing with UTF-8 encoding. I figured "everyone knows about UTF-8 right?" I retired the question pretty quickly once I realized that bit manipulation is not actually something you have to be even remotely familiar with in order to be a functional developer today in the vast majority of programming tasks, and therefore it wasn't even a good fizzbuzz-type test.
Yeah, I have an in-depth knowledge of the imperative DOM APIs, but I can't remember the last time I actually needed to use them in my day job. React provides a very good abstraction that one doesn't need to break out of very often for a lot of work.
Only 3 candidates were able to pass this code filter. The people that did pass either did well enough to pass or extremely excellent. The people that failed, maybe 19 of 22 people, all failed epically.
In practice I highly doubt that the population of candidates is actually this bimodal. It might lean heavily below the hiring bar, but I have trouble believing that it's bimodal. My own personal experience has been that it looks a lot like a normal distribution where the hiring bar is just two standard deviations above the mean (which would imply that you give offers to about 1 out of 25 people who apply, which sounds approximately in the right ballpark).
As a consequence, if your question is producing bimodal results, I can't help but wonder whether it's a good interview question, because knowledge tests exhibit much of the same type of distribution of outcomes.
In my experience, if you give the candidate only one hour, access to documentation only helps with recalling past experience, because one hour is too much time pressure to actually calm down, pick through, and digest documentation on something you haven't seen before. If you knew the DOM functions existed and just needed their names, it'll work, but if you're unfamiliar with what the DOM already offers, even if you could learn it fine on the job, you won't learn it fine under time pressure. I recently adjusted one of my interview questions specifically to reduce the need for looking up documentation because I noticed exactly this phenomenon.