Identifying Automatically Generated Headlines using Transformers

TL;DR

This paper demonstrates that transformer-based models can effectively distinguish between human-written and AI-generated headlines, achieving high accuracy and aiding in combating misinformation online.

Contribution

The study introduces a new dataset of human and AI-generated headlines and shows that transformers outperform humans in identifying fake headlines.

Findings

Transformers achieved 85.7% accuracy in detection.

Humans only identified 47.8% of fake headlines.

A new dataset for AI-generated headline detection was created.

Abstract

False information spread via the internet and social media influences public opinion and user activity, while generative models enable fake content to be generated faster and more cheaply than had previously been possible. In the not so distant future, identifying fake content generated by deep learning models will play a key role in protecting users from misinformation. To this end, a dataset containing human and computer-generated headlines was created and a user study indicated that humans were only able to identify the fake headlines in 47.8% of the cases. However, the most accurate automatic approach, transformers, achieved an overall accuracy of 85.7%, indicating that content generated from language models can be filtered out accurately.