Seeing Through AI's Lens: Enhancing Human Skepticism Towards LLM-Generated Fake News

TL;DR

This paper introduces a new metric called ESAS to help people distinguish between human-written and LLM-generated news articles, aiming to increase skepticism towards fake news created by AI.

Contribution

The study proposes the Entropy-Shift Authorship Signature (ESAS), a novel metric based on information theory, and demonstrates its effectiveness in identifying LLM-generated fake news.

Findings

High accuracy in detecting LLM-generated news using ESAS and simple classifiers.

A dataset of 39,000 news articles from human and LLM sources was compiled.

Top ESAS-ranked terms can effectively guide skepticism towards fake news.

Abstract

LLMs offer valuable capabilities, yet they can be utilized by malicious users to disseminate deceptive information and generate fake news. The growing prevalence of LLMs poses difficulties in crafting detection approaches that remain effective across various text domains. Additionally, the absence of precautionary measures for AI-generated news on online social platforms is concerning. Therefore, there is an urgent need to improve people's ability to differentiate between news articles written by humans and those produced by LLMs. By providing cues in human-written and LLM-generated news, we can help individuals increase their skepticism towards fake LLM-generated news. This paper aims to elucidate simple markers that help individuals distinguish between articles penned by humans and those created by LLMs. To achieve this, we initially collected a dataset comprising 39k news articles authored by humans or generated by four distinct LLMs with varying degrees of fake. We then devise a metric named Entropy-Shift Authorship Signature (ESAS) based on the information theory and entropy principles. The proposed ESAS ranks terms or entities, like POS tagging, within news articles based on their relevance in discerning article authorship. We demonstrate the effectiveness of our metric by showing the high accuracy attained by a basic method, i.e., TF-IDF combined with logistic regression classifier, using a small set of terms with the highest ESAS score. Consequently, we introduce and scrutinize these top ESAS-ranked terms to aid individuals in strengthening their skepticism towards LLM-generated fake news.