Generalization Gaps in Political Fake News Detection: An Empirical Study on the LIAR Dataset

TL;DR

This study systematically evaluates machine learning models on the LIAR dataset, revealing a performance ceiling and significant generalization gaps, emphasizing the need for external knowledge in political fake news detection.

Contribution

It provides a diagnostic analysis showing that increasing model complexity alone does not improve political fake news detection, highlighting the importance of external knowledge integration.

Findings

Models hit a performance ceiling with F1-score around 0.32.

Simple linear SVM matches transformer performance, indicating limited gains from complexity.

Tree-based models overfit training data and fail to generalize, relying on lexical memorization.

Abstract

The proliferation of linguistically subtle political disinformation poses a significant challenge to automated fact-checking systems. Despite increasing emphasis on complex neural architectures, the empirical limits of text-only linguistic modeling remain underexplored. We present a systematic diagnostic evaluation of nine machine learning algorithms on the LIAR benchmark. By isolating lexical features (Bag-of-Words, TF-IDF) and semantic embeddings (GloVe), we uncover a hard "Performance Ceiling", with fine-grained classification not exceeding a Weighted F1-score of 0.32 across models. Crucially, a simple linear SVM (Accuracy: 0.624) matches the performance of pre-trained Transformers such as RoBERTa (Accuracy: 0.620), suggesting that model capacity is not the primary bottleneck. We further diagnose a massive "Generalization Gap" in tree-based ensembles, which achieve more than 99% training accuracy but collapse to approximately 25% on test data, indicating reliance on lexical memorization rather than semantic inference. Synthetic data augmentation via SMOTE yields no meaningful gains, confirming that the limitation is semantic (feature ambiguity) rather than distributional. These findings indicate that for political fact-checking, increasing model complexity without incorporating external knowledge yields diminishing returns.