To start, let me say that I would never do hiring this way now. It was a flawed way to do hiring.
We ended up interviewing many of them. They were all very interested in the job. In most cases it was their second or third job out of college. ie. not junior engineers.
We got some great engineers out of it. In fact I think most of them are still there a decade later.
> To start, let me say that I would never do hiring this way now. It was a flawed way to do hiring.
I'm curious on why you view it as flawed. At a previous place I worked we had something like that (we had a small puzzle in the code version of the homepage source).
We treated it as one of the several "acquisition channels" we had for our funnel (recruiters, job-boards and job fairs were others). All channels were treated the same with regards to evaluation, so for us it was just an additional way to get candidates.
We ended up interviewing many of them. They were all very interested in the job. In most cases it was their second or third job out of college. ie. not junior engineers.
We got some great engineers out of it. In fact I think most of them are still there a decade later.