The last time time the opinion polls were so wrong was in 1992, when similarly they overestimated support for a somewhat awkward Labour leader who had been the subject of a massive negative campaign by the popular press. I suspect that this kind of personal campaign had an effect when people came to the morning of the election and thought 'actually, do I really want this guy to be Prime Minister?'.
Note that the same polls were all pretty accurate in regards to the result in Scotland, which makes the errors even odder (and gives a bit of support to the "shy conservative" theory).
Possibly the highly promoted guides to tactical voting provided by the right-wing press had a disproportionate impact on the marginal seats?
How does the "shy conservative" come into play in telephone polls, for example? I guess it may well, but it seems a little strange.
Is there something where on the day people swing more towards the safety of the incumbent? This study seems to suggest the opposite should true [0], at least when it comes to undecided voters.
It's definitely an interesting question. How were the polls so wrong? Maybe in the closely contested seats the minor parties split labour? But why didn't the polls show that?
I'd love to paw through the data. Really need to look at it on an electorate by electorate basis to see the variation between the parties (opinion vs actual).
The basic theory is that people are embarrassed to vote in their own selfish self-interest. There are some who claim you get more conservative voters with online polls, because there's no human being there to judge them.
I'd always heard that the undecided break towards the incumbent. I believe all the polls took that into account already though, and still got it wrong.
Oddly enough I would rather not have someone patronizingly voting for what they believe is in my best interests. How do they know what that might be? I would hope that people would always vote in their own 'selfish' self-interest.
I might vote for something that's not directly in my self-interest, believing it's more in the collective interest. I'm part of the collective and I feel a more even society is in my interests, even if that means I personally might not be as well off as I could be. That's really not patronizing.
Though you could argue that I've done it for my own self-interests in the long term (who knows when you or your kids might rely on essential services) :)
EDIT Also, it's pretty obvious that there are choices that are better for some groups of people (that aren't me). I don't know who "you" are, but there are voting choices that I can make that will help those classes of people, you may or may not be among them. I resent the idea that I'm being patronizing by making that choice (though in the strictest sense of the word "patron" maybe you're right).
'Collective' suggests to me we're not going to agree. Patronizing in this context only means making assumptions (likely to be unwarranted I’d argue) about what someone else wants. But who is the 'someone' affected by a vote in a general election? How many people? To what extent? What about unknown/known effects on other people who you do not have in mind? You cannot know and the assumptions pile up. So much easier to vote on what you really do know to the extent that you can know or rationalize. As you say, we can surely agree that it isn’t always our immediate concerns that best serve our interests as in the simple example of imagining ourselves out of work or needing medical attention. As you say, this benefits everyone – point being that we also are ‘everyone’.
There was discussion about the variance between phone and internet polls before the result, but the differences between the two types of polls were minute compared with the change in the actual results.
Tactical voting in an opinion poll doesn't make any difference, so I'd guess there's a danger of misleading answers. I'm sure the pollsters are at least aware of the issue, though what they can do about it I'm not sure.
It's certainly not a sample size problem. I trust the pollsters understand statistics, and this seems to be the most polled election ever.