Prefix Probing: Lightweight Harmful Content Detection for Large Language Models

TL;DR

Prefix Probing offers a fast, cost-effective black-box method for harmful content detection in large language models, matching external safety models' effectiveness without additional model deployment.

Contribution

The paper introduces Prefix Probing, a novel lightweight detection method that uses prefix comparison and caching to efficiently identify harmful content in language models.

Findings

Achieves detection accuracy comparable to external safety models.

Requires only a single log-probability computation per inference.

Operates with minimal computational overhead and no extra model deployment.

Abstract

Large language models often face a three-way trade-off among detection accuracy, inference latency, and deployment cost when used in real-world safety-sensitive applications. This paper introduces Prefix Probing, a black-box harmful content detection method that compares the conditional log-probabilities of "agreement/execution" versus "refusal/safety" opening prefixes and leverages prefix caching to reduce detection overhead to near first-token latency. During inference, the method requires only a single log-probability computation over the probe prefixes to produce a harmfulness score and apply a threshold, without invoking any additional models or multi-stage inference. To further enhance the discriminative power of the prefixes, we design an efficient prefix construction algorithm that automatically discovers highly informative prefixes, substantially improving detection performance. Extensive experiments demonstrate that Prefix Probing achieves detection effectiveness comparable to mainstream external safety models while incurring only minimal computational cost and requiring no extra model deployment, highlighting its strong practicality and efficiency.