Real-time Factuality Assessment from Adversarial Feedback
Sanxing Chen, Yukun Huang, Bhuwan Dhingra

TL;DR
This paper introduces a new evaluation method for factuality in news detection that uses adversarial feedback and retrieval-augmented generation to create challenging deceptive news variants, exposing vulnerabilities in current detectors.
Contribution
It proposes a novel pipeline leveraging RAG-based feedback to generate adversarial news examples, improving the robustness of factuality assessments for real-time news detection.
Findings
Iterative rewriting reduces detector ROC-AUC by 17.5%
RAG-based feedback enhances detection of deceptive news
Retrieval-free detectors are vulnerable to unseen events
Abstract
We show that existing evaluations for assessing the factuality of news from conventional sources, such as claims on fact-checking websites, result in high accuracies over time for LLM-based detectors-even after their knowledge cutoffs. This suggests that recent popular false information from such sources can be easily identified due to its likely presence in pre-training/retrieval corpora or the emergence of salient, yet shallow, patterns in these datasets. Instead, we argue that a proper factuality evaluation dataset should test a model's ability to reason about current events by retrieving and reading related evidence. To this end, we develop a novel pipeline that leverages natural language feedback from a RAG-based detector to iteratively modify real-time news into deceptive variants that challenge LLMs. Our iterative rewrite decreases the binary classification ROC-AUC by an absolute…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMisinformation and Its Impacts · Advanced Malware Detection Techniques · Spam and Phishing Detection
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Dropout · Byte Pair Encoding · Dense Connections · Layer Normalization · Residual Connection · Linear Warmup With Linear Decay · Attention Is All You Need · BART · Weight Decay
