FakeNewsLab: Experimental Study on Biases and Pitfalls Preventing us from Distinguishing True from False News

TL;DR

This study investigates human biases and pitfalls in distinguishing true from false news on social media, revealing counterintuitive findings about information presentation and crowd wisdom that impact fake news detection efforts.

Contribution

It provides empirical evidence on how presentation and social factors influence human accuracy in fake news identification, challenging assumptions in AI-based detection strategies.

Findings

False news is identified more accurately than true news.

Showing full articles does not improve accuracy.

Crowd wisdom and fact-checking activity enhance classification performance.

Abstract

Misinformation posting and spreading in Social Media is ignited by personal decisions on the truthfulness of news that may cause wide and deep cascades at a large scale in a fraction of minutes. When individuals are exposed to information, they usually take a few seconds to decide if the content (or the source) is reliable, and eventually to share it. Although the opportunity to verify the rumour is often just one click away, many users fail to make a correct evaluation. We studied this phenomenon with a web-based questionnaire that was compiled by 7,298 different volunteers, where the participants were asked to mark 20 news as true or false. Interestingly, false news is correctly identified more frequently than true news, but showing the full article instead of just the title, surprisingly, does not increase general accuracy. Also, displaying the original source of the news may contribute to mislead the user in some cases, while a genuine wisdom of the crowd can positively assist individuals' ability to classify correctly. Finally, participants whose browsing activity suggests a parallel fact-checking activity, show better performance and declare themselves as young adults. This work highlights a series of pitfalls that can influence human annotators when building false news datasets, which in turn fuel the research on the automated fake news detection; furthermore, these findings challenge the common rationale of AI that suggest users to read the full article before re-sharing.