I'd rather give this a shot than just shut it down with the cynical negativity that we oft see on HN. There are obvious faults like you have pointed out, but we have to try something to get us out of the the situation we're in.
This is the first honest effort I've seen that has some credibility.
There's many efforts which have credibility, and just because it has jimmy wales name on it doesn't make it credible.
Heres a previous discussion on the "fake news" problem with a far more credible start, because theyve actually got a better idea of the problem, and so are targetting an achievable goal -
Which is a conclusion shared to an extent by the fake challenge, as stated on the front page I linked
>Assessing the veracity of a news story is a complex and cumbersome task, even for trained experts [3].
>Fortunately, the process can be broken down into steps or stages. A helpful first step towards identifying fake news is to understand what other news organizations are saying about the topic. We believe automating this process, called Stance Detection, could serve as a useful building block in an AI-assisted fact-checking pipeline. So stage #1 of the Fake News Challenge (FNC-1) focuses on the task of Stance Detection.
In a nutshell, this interaction is the problem with the news cycle, and its the audience as much as its the reporting. Most people don't click on the link.
---
From the FAQ, lower down the page -
> WHY DID YOU CHOOSE THE STANCE DETECTION TASK RATHER THAN THE TASK OF LABELING A CLAIM, HEADLINE OR STORY TRUE/FALSE, WHICH SEEMS TO BE WHAT THE FAKE NEWS PROBLEM IS ALL ABOUT?
ANSWER:
There are several reasons Stance Detection makes for a good first task for the Fake News Challenge:
Our extensive discussions with journalists and fact checkers made it clear both how difficult “truth labeling” of claims really is, and how they’d rather have reliable semi-automated tool to help them in do their job better rather than fully-automated system whose performance will inevitably fall far short of 100% accuracy.
Truth labeling also poses several large technical / logistical challenge for a contest like the FNC:
There exists very little labeled training data of fake vs. real news stories.
The data that does exist (e.g. fact checker website archives) is almost all copyright protected.
The data that does exist is extremely diverse and unstructured, making hard to train on.
Any dataset containing claims with associated “truth” labels is going to be contested as biased.
Together these make the truth labeling task virtually impossible with existing AI / NLP. In fact, even people have trouble distinguishing fake news from real news.
The dataset we are using to support the Stance Detection task for FNC-1 was created by accredited journalists, making it both high quality and credible. It is also in the public domain.
Variants of the FNC-1 Stance Detection task have already been explored and proven feasible but far from trivial by Andreas Vlachos & his students from U. of Sheffield. Cite: Ferreira & Vlachos (2016) & Augenstein et al. (2016).
We considered targeting the truth labeling task for the FNC-1, but without giving teams any labeled training data. We decided against it both because we thought a competition with a more traditionally structured Machine learning tasks would appeal to more teams, and because such an open-ended truth labeling competition was recently completed, called the Fast & Furious Fact Check Challenge.
Our discussions with human fact checkers lead us to believe that a solution to the stance detection problem could form the basis of a useful tool for real-life human fact checkers. Also see, next question/answer.
thats a substantially more thought out approach to the problem than Wikitribune.
This is the first honest effort I've seen that has some credibility.