Synthetic Disinformation Attacks on Automated Fact Verification Systems

TL;DR

This paper investigates how automated fact-checking systems are vulnerable to synthetic disinformation attacks, revealing significant performance drops when faced with fabricated or altered evidence sources, highlighting emerging threats from advanced NLP generators.

Contribution

It introduces two novel adversarial attack scenarios on fact-checkers and demonstrates their effectiveness across multiple models and benchmarks, emphasizing the need for more robust verification methods.

Findings

Fact-checkers' performance drops significantly under synthetic evidence attacks.

Both fabricated and modified evidence sources can deceive automated fact-checkers.

Modern NLG systems pose a growing threat as generators of disinformation.

Abstract

Automated fact-checking is a needed technology to curtail the spread of online misinformation. One current framework for such solutions proposes to verify claims by retrieving supporting or refuting evidence from related textual sources. However, the realistic use cases for fact-checkers will require verifying claims against evidence sources that could be affected by the same misinformation. Furthermore, the development of modern NLP tools that can produce coherent, fabricated content would allow malicious actors to systematically generate adversarial disinformation for fact-checkers. In this work, we explore the sensitivity of automated fact-checkers to synthetic adversarial evidence in two simulated settings: AdversarialAddition, where we fabricate documents and add them to the evidence repository available to the fact-checking system, and AdversarialModification, where existing evidence source documents in the repository are automatically altered. Our study across multiple models on three benchmarks demonstrates that these systems suffer significant performance drops against these attacks. Finally, we discuss the growing threat of modern NLG systems as generators of disinformation in the context of the challenges they pose to automated fact-checkers.