WISE: Web Information Satire and Fakeness Evaluation

TL;DR

This paper introduces WISE, a framework for evaluating lightweight transformer models in distinguishing fake news from satire, demonstrating that small models can perform comparably to larger ones in misinformation detection.

Contribution

The study benchmarks eight lightweight transformer models on a balanced dataset, revealing their effectiveness and efficiency in fake news and satire classification tasks.

Findings

MiniLM achieved 87.58% accuracy.

RoBERTa-base achieved 95.42% ROC-AUC.

DistilBERT offers a strong efficiency-accuracy balance.

Abstract

Distinguishing fake or untrue news from satire or humor poses a unique challenge due to their overlapping linguistic features and divergent intent. This study develops WISE (Web Information Satire and Fakeness Evaluation) framework which benchmarks eight lightweight transformer models alongside two baseline models on a balanced dataset of 20,000 samples from Fakeddit, annotated as either fake news or satire. Using stratified 5-fold cross-validation, we evaluate models across comprehensive metrics including accuracy, precision, recall, F1-score, ROC-AUC, PR-AUC, MCC, Brier score, and Expected Calibration Error. Our evaluation reveals that MiniLM, a lightweight model, achieves the highest accuracy (87.58%) among all models, while RoBERTa-base achieves the highest ROC-AUC (95.42%) and strong accuracy (87.36%). DistilBERT offers an excellent efficiency-accuracy trade-off with 86.28\% accuracy and 93.90\% ROC-AUC. Statistical tests confirm significant performance differences between models, with paired t-tests and McNemar tests providing rigorous comparisons. Our findings highlight that lightweight models can match or exceed baseline performance, offering actionable insights for deploying misinformation detection systems in real-world, resource-constrained settings.