Illusions of Relevance: Arbitrary Content Injection Attacks Deceive Retrievers, Rerankers, and LLM Judges

TL;DR

This paper demonstrates that search systems and LLM relevance judges are highly vulnerable to content injection attacks, which can manipulate search rankings and relevance scores, raising concerns about robustness and trustworthiness.

Contribution

It introduces a comprehensive analysis of how arbitrary content injection attacks deceive retrieval and ranking models, revealing vulnerabilities across various model types and sizes.

Findings

Retrievers, rerankers, and LLM judges are highly susceptible to content injection.

Injection success depends on factors like model class, size, and content toxicity.

Current defenses often fail to detect injected content.

Abstract

This work considers a black-box threat model in which adversaries attempt to propagate arbitrary non-relevant content in search. We show that retrievers, rerankers, and LLM relevance judges are all highly vulnerable to attacks that enable arbitrary content to be promoted to the top of search results and to be assigned perfect relevance scores. We investigate how attackers may achieve this via content injection, injecting arbitrary sentences into relevant passages or query terms into arbitrary passages. Our study analyzes how factors such as model class and size, the balance between relevant and non-relevant content, injection location, toxicity and severity of injected content, and the role of LLM-generated content influence attack success, yielding novel, concerning, and often counterintuitive results. Our results reveal a weakness in embedding models, LLM-based scoring models, and generative LLMs, raising concerns about the general robustness, safety, and trustworthiness of language models regardless of the type of model or the role in which they are employed. We also emphasize the challenges of robust defenses against these attacks. Classifiers and more carefully prompted LLM judges often fail to recognize passages with content injection, especially when considering diverse text topics and styles. Our findings highlight the need for further research into arbitrary content injection attacks. We release our code for further study.