Exploring the Potential of the Large Language Models (LLMs) in Identifying Misleading News Headlines

TL;DR

This study evaluates the effectiveness of Large Language Models like ChatGPT-3.5, ChatGPT-4, and Gemini in detecting misleading news headlines across various domains, highlighting ChatGPT-4's superior performance and the importance of human-centered evaluation.

Contribution

It provides a comparative analysis of LLMs in misinformation detection, emphasizing the role of human judgment and ethical considerations in model development.

Findings

ChatGPT-4 outperforms other models in accuracy.

Model performance varies significantly across headlines.

Human evaluation is crucial for nuanced misinformation detection.

Abstract

In the digital age, the prevalence of misleading news headlines poses a significant challenge to information integrity, necessitating robust detection mechanisms. This study explores the efficacy of Large Language Models (LLMs) in identifying misleading versus non-misleading news headlines. Utilizing a dataset of 60 articles, sourced from both reputable and questionable outlets across health, science & tech, and business domains, we employ three LLMs- ChatGPT-3.5, ChatGPT-4, and Gemini-for classification. Our analysis reveals significant variance in model performance, with ChatGPT-4 demonstrating superior accuracy, especially in cases with unanimous annotator agreement on misleading headlines. The study emphasizes the importance of human-centered evaluation in developing LLMs that can navigate the complexities of misinformation detection, aligning technical proficiency with nuanced human judgment. Our findings contribute to the discourse on AI ethics, emphasizing the need for models that are not only technically advanced but also ethically aligned and sensitive to the subtleties of human interpretation.