If you allow people to vote after they have seen the rankings, you have the wisdom of mobs. The only way to get the wisdom of crowds is to ask people before they have seen other people's opinions.
I wonder what would happen on this site if the comments and stories were still sorted by points but the actual number was hidden (or perhaps only hidden until after you vote but I don't see why they need to be there period).
That would completely change how I vote. I vote by comparing a post's score to the amount I would give it if I could pick a value. If they're significantly different, I vote accordingly.
If you're really curious checkout Perlmonks.org. It's been around for a decade and has consistently maintained a community that more closely resembles this site than any other I've seen. Blind voting has been in use there for a very long time and has worked quite well. You're also alloted a limited number of points per day based on your karma (XP) level.
>> (or perhaps only hidden until after you vote but I don't see why they need to be there period)
That's an interesting point--why does every site need to quantify for the user the particular score of a comment? Why not indicate popularity by simply ordering the comments?
For example color-shading. While it may still be possible to re-extract numbers from the colors --- psychological colors may have another effect than numbers.
It seems like in this case we have a temporal error. A popular movie generated buzz now and skewed a result that people pronounce loosely as of all time.
Batman might be the best movie out recently but in 2 years time, it will be a dim and fading memory while the Godfather will retain its appeal as it has already proven that it can.
IMDB just needs a temporal filter to keep the latest trend from skewing the universal result.
I think that the problem is similar to the eternal September problem. I'm sure that the people who had been voting and contributing to the ratings on IMDB had been involved on the site for a while. Now, there's a big influx of people on IMDB that are voting for their favorite movie. I bet that many of the people who are voting Dark Knight up have never seen the Godfather or Shawshank.
Same thing happened to my wife in film school. She went to film school because she loved the classics of the 30's - 50's. She was stunned to find out that the kids she was going to school with thought that Austin Powers was the best film ever.
This mob mentality, brought on by a significant event (in this case, the release of The Dark Knight), causes a total breakdown in a system that is otherwise fairly accurate, useful, and trustworthy.
I wouldn't call this a "total breakdown in the system". I would call it a minor blip. I mean, ultimately the system recommends what movies to watch. If someone isn't going to watch the Godfather now that it's ranked #3 instead of #1, then the person is flawed, not the system.
Otherwise, this was an interesting post. I have been astonished that TDK is #1. Didn't realize people were also downvoting other movies to keep it there.
The flaw in the system isn't that The Godfather was bumped down. The flaw is that people are rating the Godfather down to the point at which it changed its ranking with respect to other movies.
And you're right, it's not a major flaw for a movie ranking system. But let's suppose something similar to the IMDB system(which would be an approval/range voting system) is used in an election, and the first place gets President, the second choice getting Veep. With an election, you don't have real-time rankings like IMDB, but you have something similar--polls. And that's even worse, since there's a few points of inaccuracy. Strategically worded polls could change the results of an election.
But that change is a blip in and of itself. Godfather has been bumped down before. Happened with Lord of the Rings. It climbs back up after the initial wave.
Score Voting (aka Range Votiing) and Approval Voting are actually the best voting methods, especially when you take strategic voting into account.
http://rangevoting.org/StratHonMix.html
In this case, even if the election result changed, you'd still get a candidate who was rated relatively well by the average voter, so the results would not be that much worse. Now consider that other voting methods react much worse to strategic voters, _and_ that score voting is as good with a great number of strategic voters as other systems are with 100% honest voters, and this whole argument falls flat.
If it's a minor blip, how do you propose they now fix the problem? I see it is as so systemic and flawed that the only solution is to scrap it altogether.
My point is you don't. The system is still working fine. If Godfather sank to #300, then there would be something to fix. Even with all of the shenanigans of Dark Knight fans, the result on Godfather's rankings is a minuscule blip that could be explained by chance and should have no effect on whether someone thinks Godfather is a movie worth watching.
Have they ruled out the possibility that the people marketing the movie are gaming the system? Even if voting was restricted to established IMDB users, surely a reasonably well connected Hollywood marketing person could round up a few hundred of those among friends and friends of friends.
Hmm, those numbers are definitely too big to be influenced by anyone's network. Judging from the other top-ranked films, it looks as if IMDB's userbase is in the middle of changing.
I don't think they need to game the system. The lukewarm or negative reviews on RottenTomatoes are being bombarded with genuine fanboy vitriol. (Many of the commenters, at least on the first page of comments, have joining dates that suggest they have been on RT for a while)
http://www.rottentomatoes.com/m/the_dark_knight/?critic=crea...
It is quite amusing in a sad way actually, seeing them trash the likes of David Denby and Joe Morgenstern.
IMDB's ranking system is one of the least sophisticated out there, which is surprising given that they're owned by Amazon these days. I assume it's still just the same couple of guys that started it left to run it in relative independence. You really don't need to game their system much; it's just bad technology.
However, if "The Dark Knight" bumping off "The Godfather" was all that was happening, that wouldn't be a problem.
What's happening, though, is that the release of "The Dark Knight" made "The Shawshank Redemption" have a higher rank than "The Godfather". This means that the ranking system has lead to the rating system being used strategically.
The implications of this for approval voting would be an interesting corollary. The polls could influence themselves.
I've seen The Godfather. I've seen The Dark Knight. Now, maybe I'm just missing the point of that Godfather thing, but I'd say The Godfather had more dramatic tension and The Dark Knight had more philosophical depth.
No, stop laughing, I'm serious. Tell me what point The Godfather makes.
Blockbusters popular with the avid Internet users have always started at incredibly high ratings only to slowly fally to the right place. The LOTR movies were in top-10 once too.
I suppose IMDB might try to account for the initial voting wave being raving fans and dampen the initial votes, but why would they? Personally I've reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still at #1.
The most popular topic on the IMDB message boards is "This does not deserve to be in top 250" or "Why is this not in Top 250 why some-other-movie is not?". People like to keep an eye on rankings and to comment on them.
The main practical use for the ratings IMHO is to get a list of movies within each genre sorted by the rating, to see if you missed something people thought was great.
exactly -- imdb has an internet and populist skew. As long as you take this into account, their ratings are fine and predictable. Gaming rankings is practically an official internet pastime, so it's not unexpected that this event is taking place, although I'm surprised it built up enough steam without (presumedly) official co-ordination to drop the Godfather a few places -- that is notable. I'm sure politicians are taking notes as we speak.
If you think of this system as a game and in this instance a number of users have figured out a way to act within the rules of the game to achieve an outcome inconsistent with IMDB's intended one, then IMDB can simply change the rules to get what they think is best!
It wouldn't be too difficult to observe the behavior of the offending members (vote for TDK, within x hours vote down Godfather...) and simply setup criteria to disallow these actions. You could also set up an algorithm to detect screwy behavior to alert a developer or block that action for a duration.
And hey, we throw error messages for repeated button clicks and such so that click-happy users or spam bots don't take down the system...can't that kind of precaution be extended if necessary? "We know you're in love with TDK, but do you really hate The Godfather...come back in a month if you still do!"
The problem is that people discuss these concepts without reading the book and thus miss out on the subtleties. The long tail is another common victim.
I don't see why it wouldn't be possible to build into the system some of the same logic that obviously makes it clear to everyone that there is a problem with these votes.
A computer might not be able to appreciate the goodness of The Godfather, but it can be taught that a movie that has consistently been voted great for decades, does not suddenly fall out of favor in a month.
The problem is not the Dark Knight fans, so much as the algorithms being used to rank the movies. How can you take advantage of crowd wisdom if you don’t control for spoofing, spam and mob voting? It’s surprising that there wasn’t already in place a process for dealing with such a common phenomenon.
I'm much less worried about that happening here because of the transient nature of the site. I don't think potential manipulators would find it worthwhile.
The point of top-k rankings is to put items in the proper order. As far as the crowds are concerned, this is no failure. Sorry your favorite movie lost its spot.
over time it should return to a 'fair' position, once temporary disequilibrium in the 'crowd' (induced by marketing, hype, whatever) is resolved. It's a shame people vote 'strategically', not to reflect their true sentiments - but whatever. Over time, with a diverse enough crowd, 'wisdom of the crowd' systems are effective and reasonably unbiased.
See, but it's all about opinion. Some people honestly like Dark Knight better, because it's far more gripping and easier to get into. And those people have every RIGHT to give Godfather a 1, if people that gave Godfather a 10 are giving The Dark Knight a 1 in return. The Godfather people started this days before the Dark Knight people came in.
This reminds me of the recent articles about the netflix optimization challenge. Voting systems like imdb might be improved with some simple heuristics that take into account the absurdity of some people's votes.
If a user makes votes that are < 2.0 for a movie that the entire community for years has rated above 9.0, then probably that user's votes should all be ignored, because they are totally uninformed.
What if you assume everything is average and you give more weight toward the finally rating for those individual ratings that are further from the norm? This means people couldn't game it by following the crowd, following the crowd gives the entire crowds rating less weight.
