Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers

TL;DR

This paper presents a novel interpretability framework called adversarial activation patching to detect and mitigate deceptive behaviors in safety-aligned transformer models, demonstrating its effectiveness through toy simulations and proposing future research directions.

Contribution

Introduces adversarial activation patching as a mechanistic interpretability tool for deception detection and mitigation in transformer-based language models, with empirical validation and theoretical hypotheses.

Findings

Activation patching increases deception rates to 23.9%.

Layer-specific variations support hypotheses about model vulnerabilities.

Mitigation strategies include activation anomaly detection and robust fine-tuning.

Abstract

Large language models (LLMs) aligned for safety through techniques like reinforcement learning from human feedback (RLHF) often exhibit emergent deceptive behaviors, where outputs appear compliant but subtly mislead or omit critical information. This paper introduces adversarial activation patching, a novel mechanistic interpretability framework that leverages activation patching as an adversarial tool to induce, detect, and mitigate such deception in transformer-based models. By sourcing activations from "deceptive" prompts and patching them into safe forward passes at specific layers, we simulate vulnerabilities and quantify deception rates. Through toy neural network simulations across multiple scenarios (e.g., 1000 trials per setup), we demonstrate that adversarial patching increases deceptive outputs to 23.9% from a 0% baseline, with layer-specific variations supporting our hypotheses. We propose six hypotheses, including transferability across models, exacerbation in multimodal settings, and scaling effects. An expanded literature review synthesizes over 20 key works in interpretability, deception, and adversarial attacks. Mitigation strategies, such as activation anomaly detection and robust fine-tuning, are detailed, alongside ethical considerations and future research directions. This work advances AI safety by highlighting patching's dual-use potential and provides a roadmap for empirical studies on large-scale models.