> Returns start diminishing very quickly after the first five minutes.
That's the opposite of a scientific approach. One of the best things I ever learned from an experienced interviewer is the spend the rest of the interview trying to disprove your first impression. Anything else is an exercise in confirmation bias. I'm not saying that I've never ended an interview early when it's obvious there is a bad fit. My time is important and so is other people's time.
A scientific approach is to pose a theory (this candidate would make a good hire) and perform a measurement (whoa, this guy's good) and then perhaps perform the measurement again, without bias, to see if the measurement is consistent and reproducible.
What you're suggesting is to spend the next several hours performing additional measurements in the hopes of finding one contrary to your first measurement. That is precisely the opposite of science. It's also how casinos stay in business.
And that's exactly what I said. Don't trust your initial impression. Measure again. Get as many data points as you can.
> What you're suggesting is to spend the next several hours
I didn't suggest that and I'm not sure where you got that idea from. My interviews are usually less than an hour and then I hand the candidate over to the next interviewer. If they are a strong no from me, I'll even talk to the next interviewer to explain why and see if they even want to continue with the candidate. They usually don't want to continue with a "strong no" candidate, but sometimes the next interviewer might not share my concerns and they do want to continue with the candidate. Could be the person is a good fit for the other interviewer's team.
That's the opposite of a scientific approach. One of the best things I ever learned from an experienced interviewer is the spend the rest of the interview trying to disprove your first impression. Anything else is an exercise in confirmation bias. I'm not saying that I've never ended an interview early when it's obvious there is a bad fit. My time is important and so is other people's time.