The SMeL Test: A simple benchmark for media literacy in language models

TL;DR

The paper introduces the SMeL Test, a benchmark to evaluate language models' ability to filter untrustworthy online content, revealing current models' limitations in media literacy and hallucination issues.

Contribution

It presents the SMeL Test as a minimal benchmark for media literacy in LLMs and evaluates various models, highlighting their shortcomings in filtering misinformation.

Findings

No model consistently filters untrustworthy content.

Reasoning models perform better but still hallucinate up to 70%.

Larger models do not always outperform smaller ones.

Abstract

The internet is rife with unattributed, deliberately misleading, or otherwise untrustworthy content. Though large language models (LLMs) are often tasked with autonomous web browsing, the extent to which they have learned the simple heuristics human researchers use to navigate this noisy environment is not currently known. In this paper, we introduce the Synthetic Media Literacy Test (SMeL Test), a minimal benchmark that tests the ability of language models to actively filter out untrustworthy information in context. We benchmark a variety of commonly used instruction-tuned LLMs, including reasoning models, and find that no model consistently succeeds; while reasoning in particular is associated with higher scores, even the best API model we test hallucinates up to 70% of the time. Remarkably, larger and more capable models do not necessarily outperform their smaller counterparts. We hope our work sheds more light on this important form of hallucination and guides the development of new methods to combat it.