BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages

TL;DR

BLUFF is a large-scale multilingual benchmark dataset designed to evaluate false and synthetic content detection across 79 languages, addressing the gap in low-resource language coverage and providing tools for advancing equitable misinformation detection.

Contribution

The paper introduces BLUFF, a comprehensive multilingual benchmark dataset covering 79 languages, with novel content generation and filtering methods, to improve false content detection in low-resource languages.

Findings

State-of-the-art detectors degrade up to 25.3% F1 on low-resource languages.

BLUFF covers both high-resource and low-resource languages, filling a critical research gap.

Extensive linguistic-oriented evaluation and open-source tools are provided.

Abstract

Multilingual falsehoods threaten information integrity worldwide, yet detection benchmarks remain confined to English or a few high-resource languages, leaving low-resource linguistic communities without robust defense tools. We introduce BLUFF, a comprehensive benchmark for detecting false and synthetic content, spanning 79 languages with over 202K samples, combining human-written fact-checked content (122K+ samples across 57 languages) and LLM-generated content (79K+ samples across 71 languages). BLUFF uniquely covers both high-resource "big-head" (20) and low-resource "long-tail" (59) languages, addressing critical gaps in multilingual research on detecting false and synthetic content. Our dataset features four content types (human-written, LLM-generated, LLM-translated, and hybrid human-LLM text), bidirectional translation (English$\leftrightarrow$X), 39 textual modification techniques (36 manipulation tactics for fake news, 3 AI-editing strategies for real news), and varying edit intensities generated using 19 diverse LLMs. We present AXL-CoI (Adversarial Cross-Lingual Agentic Chainof-Interactions), a novel multi-agentic framework for controlled fake/real news generation, paired with mPURIFY, a quality filtering pipeline ensuring dataset integrity. Experiments reveal state-of-theart detectors suffer up to 25.3% F1 degradation on low-resource versus high-resource languages. BLUFF provides the research community with a multilingual benchmark, extensive linguistic-oriented benchmark evaluation, comprehensive documentation, and opensource tools to advance equitable falsehood detection. Dataset and code are available at: https://jsl5710.github.io/BLUFF/