PhantomLint: Principled Detection of Hidden LLM Prompts in Structured Documents

TL;DR

PhantomLint is a new tool that detects hidden LLM prompts in structured documents, helping prevent prompt injection attacks and ensuring trust in AI-assisted document processing systems.

Contribution

This paper introduces a principled approach and prototype tool for detecting hidden LLM prompts in structured documents, a novel contribution in AI security.

Findings

High applicability across diverse document types

Very low false positive rate (~0.092%)

Effective detection of various hiding methods

Abstract

Hidden LLM prompts have appeared in online documents with increasing frequency. Their goal is to trigger indirect prompt injection attacks while remaining undetected from human oversight, to manipulate LLM-powered automated document processing systems, against applications as diverse as r\'esum\'e screeners through to academic peer review processes. Detecting hidden LLM prompts is therefore important for ensuring trust in AI-assisted human decision making. This paper presents the first principled approach to hidden LLM prompt detection in structured documents. We implement our approach in a prototype tool called PhantomLint. We evaluate PhantomLint against a corpus of 3,402 documents, including both PDF and HTML documents, and covering academic paper preprints, CVs, theses and more. We find that our approach is generally applicable against a wide range of methods for hiding LLM prompts from visual inspection, has a very low false positive rate (approx. 0.092%), is practically useful for detecting hidden LLM prompts in real documents, while achieving acceptable performance.