RAGShield: Detecting Numerical Claim Manipulation in Government RAG Systems
KrishnaSaiReddy Patil

TL;DR
RAGShield is a novel system that detects numerical claim manipulation in government retrieval-augmented generation systems by directly analyzing extracted values, overcoming embedding-based blind spots.
Contribution
It introduces a pattern-based engine and cross-source verification approach to accurately identify manipulated numerical claims in government documents.
Findings
RAGShield detects 100% of manipulated numerical claims in tested IRS documents.
Embedding-based defenses miss 79-90% of the same attacks due to a fundamental blind spot.
RAGShield achieves high entity detection accuracy at 99.8% on real IRS passages.
Abstract
Retrieval-Augmented Generation (RAG) systems are deployed across federal agencies for citizen-facing tax guidance, benefits eligibility, and legal information, where a single incorrect number causes direct financial harm. This paper proves that all embedding-based RAG defenses share a fundamental blind spot: changing a tax deduction by $50,000 produces cosine similarity 0.9998, invisible to every known detection threshold. Across 174 manipulation pairs and two embedding models, the mean sensitivity gap is 1,459x. The blind spot is confirmed on real IRS documents.The root cause is that embeddings encode topic, not numerical precision. RAGShield sidesteps this by operating on extracted values directly: a pattern-based engine identifies dollar amounts and percentages in government text, links each value to its governing entity through two-pass context propagation (99.8% entity detection on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
