Are LLMs More Skeptical of Entertainment News?

TL;DR

This study investigates whether large language models exhibit genre-based biases in news credibility assessment, finding that some models are more prone to misclassify entertainment news as fake, with implications for evaluation practices.

Contribution

It reveals genre-specific false-positive biases in LLMs and demonstrates that prompt framing can mitigate these biases in certain models, highlighting the importance of genre-aware evaluation.

Findings

DeepSeek-V3.2 and GPT-5.2 show significant genre bias in false positives.

Style-swap experiments indicate bias is not solely due to stylistic differences.

Prompt framing reduces false positives for some models without affecting recall.

Abstract

Large language models (LLMs) are increasingly used for automated news credibility assessment, yet it remains unclear whether they apply even-handed standards across journalistic genres. We examine whether zero-shot LLMs are more likely to misclassify legitimate entertainment news as fake than legitimate hard news, using a within-dataset design on GossipCop from FakeNewsNet. Across four frontier models, we find a clear but model-specific genre asymmetry: DeepSeek-V3.2 and GPT-5.2 show false-positive-rate gaps of 10.1 and 8.8 percentage points, respectively (both $p < .001$), whereas Claude Opus 4.6 and Gemini 3 Flash show no comparable difference. A style-swap experiment yields only limited and inconsistent changes, suggesting that the asymmetry is not reducible to stylistic register alone. Prompt-based mitigation is likewise possible but not generic: framing the model as an entertainment-news fact-checker reduces false positives for DeepSeek-V3.2 by about 50\% without detectable recall loss, but offers little improvement for GPT-5.2. Exploratory qualitative coding further suggests two recurring error patterns in sampled false positives: treating private-life claims as inherently unverifiable and discounting entertainment journalism as an epistemically weaker genre. Taken together, these findings show that aggregate performance metrics can obscure structured false positives within legitimate journalism. We argue that LLM-based credibility assessment may not only evaluate truth claims but also differentially recognize the legitimacy of journalistic genres, and that evaluation should therefore include genre-stratified false-positive analysis alongside overall accuracy.