Linearly Decoding Refused Knowledge in Aligned Language Models

TL;DR

This paper demonstrates that much of the refused harmful information in aligned language models remains linearly decodable from their hidden states, indicating that instruction-tuning suppresses but does not eliminate such information.

Contribution

It reveals that refused knowledge in instruction-tuned LMs is linearly accessible and persists from base models, challenging assumptions about the effectiveness of alignment techniques.

Findings

Refused information is highly linearly decodable with correlations exceeding 0.8.

Probes trained on base models transfer to instruction-tuned models, revealing persistent internal representations.

Decoded information correlates with the models' subtle generative behaviors, indicating active use of suppressed knowledge.

Abstract

Most commonly used language models (LMs) are instruction-tuned and aligned using a combination of fine-tuning and reinforcement learning, causing them to refuse users requests deemed harmful by the model. However, jailbreak prompts can often bypass these refusal mechanisms and elicit harmful responses. In this work, we study the extent to which information accessed via jailbreak prompts is decodable using linear probes trained on LM hidden states. We show that a great deal of initially refused information is linearly decodable. For example, across models, the response of a jailbroken LM for the average IQ of a country can be predicted by a linear probe with Pearson correlations exceeding $0.8$. Surprisingly, we find that probes trained on base models (which do not refuse) sometimes transfer to their instruction-tuned versions and are capable of revealing information that jailbreaks decode generatively, suggesting that the internal representations of many refused properties persist from base LMs through instruction-tuning. Importantly, we show that this information is not merely "leftover" in instruction-tuned models, but is actively used by them: we find that probe-predicted values correlate with LM generated pairwise comparisons, indicating that the information decoded by our probes align with suppressed generative behavior that may be expressed more subtly in other downstream tasks. Overall, our results suggest that instruction-tuning does not wholly eliminate or even relocate harmful information in representation space-they merely suppress its direct expression, leaving it both linearly accessible and indirectly influential in downstream behavior.