Red-teaming Activation Probes using Prompted LLMs

TL;DR

This paper introduces a lightweight black-box red-teaming method using prompt-based feedback with LLMs to identify failure modes in activation probes, revealing vulnerabilities and brittleness patterns.

Contribution

It presents a novel, minimal-effort approach for probing AI systems' robustness without fine-tuning or access to model internals.

Findings

Identified brittleness patterns like false positives from legalese

Detected persistent vulnerabilities under scenario-constraint attacks

Provided insights for improving probe robustness

Abstract

Activation probes are attractive monitors for AI systems due to low cost and latency, but their real-world robustness remains underexplored. We ask: What failure modes arise under realistic, black-box adversarial pressure, and how can we surface them with minimal effort? We present a lightweight black-box red-teaming procedure that wraps an off-the-shelf LLM with iterative feedback and in-context learning (ICL), and requires no fine-tuning, gradients, or architectural access. Running a case study with probes for high-stakes interactions, we show that our approach can help discover valuable insights about a SOTA probe. Our analysis uncovers interpretable brittleness patterns (e.g., legalese-induced FPs; bland procedural tone FNs) and reduced but persistent vulnerabilities under scenario-constraint attacks. These results suggest that simple prompted red-teaming scaffolding can anticipate failure patterns before deployment and might yield promising, actionable insights to harden future probes.