Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks
Shengyao Zhuang, Ekaterina Khramtsova, Xueguang Ma, Bevan Koopman,, Jimmy Lin, Guido Zuccon

TL;DR
This paper demonstrates that vision-language model-based document retrievers are highly vulnerable to pixel poisoning attacks, which can significantly disrupt search results even with minimal adversarial input.
Contribution
The study introduces three pixel poisoning attack methods and empirically evaluates their effectiveness against VLM-based retrievers, revealing their high vulnerability.
Findings
Injecting a single adversarial screenshot can disrupt top-10 retrievals for nearly 42% of queries.
Vulnerabilities are more severe when attacking specific known queries.
Pixel poisoning attacks outperform similar attacks on text-only retrievers.
Abstract
Recent advancements in dense retrieval have introduced vision-language model (VLM)-based retrievers, such as DSE and ColPali, which leverage document screenshots embedded as vectors to enable effective search and offer a simplified pipeline over traditional text-only methods. In this study, we propose three pixel poisoning attack methods designed to compromise VLM-based retrievers and evaluate their effectiveness under various attack settings and parameter configurations. Our empirical results demonstrate that injecting even a single adversarial screenshot into the retrieval corpus can significantly disrupt search results, poisoning the top-10 retrieved documents for 41.9% of queries in the case of DSE and 26.4% for ColPali. These vulnerability rates notably exceed those observed with equivalent attacks on text-only retrievers. Moreover, when targeting a small set of known queries, the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital Media Forensic Detection
MethodsSparse Evolutionary Training
