A Study of Fake News Reading and Annotating in Social Media Context

TL;DR

This paper investigates how humans read and annotate fake news on social media through eye-tracking and expert analysis, aiming to improve automated detection methods and understand user behavior.

Contribution

It introduces a novel eye-tracking dataset of social media news reading and annotation, providing insights into human fake news detection processes.

Findings

Participants showed distinct reading patterns when identifying fake news.

The dataset reveals key features used by humans to detect misinformation.

Expert annotators provided high-quality labels for fake news detection.

Abstract

The online spreading of fake news is a major issue threatening entire societies. Much of this spreading is enabled by new media formats, namely social networks and online media sites. Researchers and practitioners have been trying to answer this by characterizing the fake news and devising automated methods for detecting them. The detection methods had so far only limited success, mostly due to the complexity of the news content and context and lack of properly annotated datasets. One possible way to boost the efficiency of automated misinformation detection methods, is to imitate the detection work of humans. It is also important to understand the news consumption behavior of online users. In this paper, we present an eye-tracking study, in which we let 44 lay participants to casually read through a social media feed containing posts with news articles, some of which were fake. In a second run, we asked the participants to decide on the truthfulness of these articles. We also describe a follow-up qualitative study with a similar scenario but this time with 7 expert fake news annotators. We present the description of both studies, characteristics of the resulting dataset (which we hereby publish) and several findings.