This perfectly encapsulates my unease with this kind of recruiting/self selection approach. I get that culture fit is important, but I also don't like the sort of groupthink and monoculture that you get by just hiring a bunch of people who are all into the same narrow pop culture niches.
I'm guilty of this kind of monocultural preference myself. I used to joke about starting interviews with "What's your favorite Hayao Miyazaki film and why?" although I never actually did it.
I understand your point, but just because we all love Miyazaki films (the objective truth :D) doesn't mean we each don't have our own unique traits/interests/hobbies/etc. that provide a well-rounded cultural outlook.
I agree with your notion that in hiring one should maintain a good balance of breadth and culture fit!
What percentage of 20-30 year old males do you believe have played Portal (2), saw this posting, had the inclination to download the mod, and the inquisitiveness and persistance to actually complete the puzzle/test?
Rarely is a filter perfect, but this is likely a lot better than taking a random sample from the population :)
>In this case, 20-30 year old males.
I would argue that it's more like "Anyone who reads Hacker News, Reddit, or any other place this gets posted." (Granted, that overlaps with 20-30 year old males quite a bit.) The puzzle is more like an advertisement, not a requirement to apply.
In this case, 20-30 year old males.