Factuality Checking in News Headlines with Eye Tracking

TL;DR

This study explores whether eye-tracking data can determine the truthfulness of news headlines, demonstrating that false headlines attract less visual attention and that a model can predict factuality with moderate accuracy using eye movements.

Contribution

It introduces a novel approach to fact-checking news headlines solely based on eye-tracking data, showing its effectiveness and minimal user requirements.

Findings

False headlines receive less visual attention than true ones.

The ensemble model achieves a mean AUC of 0.688 in predicting factuality.

Eye-tracking data from 25 users reading 3-6 headlines suffices for accurate predictions.

Abstract

We study whether it is possible to infer if a news headline is true or false using only the movement of the human eyes when reading news headlines. Our study with 55 participants who are eye-tracked when reading 108 news headlines (72 true, 36 false) shows that false headlines receive statistically significantly less visual attention than true headlines. We further build an ensemble learner that predicts news headline factuality using only eye-tracking measurements. Our model yields a mean AUC of 0.688 and is better at detecting false than true headlines. Through a model analysis, we find that eye-tracking 25 users when reading 3-6 headlines is sufficient for our ensemble learner.