You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases

TL;DR

This paper demonstrates that language models can covertly acquire behavioral traits from paraphrased data generated by other models, even when the content contradicts the original preferences, raising concerns for training pipelines.

Contribution

It shows that subliminal learning occurs through natural language paraphrases, regardless of semantic relation or explicit contradiction, highlighting a new challenge in model training security.

Findings

Paraphrased data can transmit model preferences up to 19 percentage points.

Content unrelated or explicitly contradictory to original traits does not prevent transmission.

Filtering for paraphrase fidelity does not stop subliminal trait transfer.

Abstract

When language models are trained on synthetic data, they (student model) can covertly acquire behavioral traits from the data-generating model (teacher model). Subliminal learning refers to the transmission of traits from a teacher to a student model via training on data unrelated to those traits. Prior work demonstrated this in the training domains of number sequences, code, and math Chain-of-Thought traces including transmission of misaligned behaviors. We investigate whether transmission occurs through natural language paraphrases with fixed semantic content, and whether content explicitly contradicting the teacher's preference can block it. We find that training on paraphrases from a teacher system-prompted to love a particular animal increases a student's preference for that animal by up to 19 percentage points. This occurs when paraphrased content is semantically unrelated to the animal, or even when it explicitly expresses dislike. The transmission succeeds despite aggressive filtering to ensure paraphrase fidelity. This raises concerns for pipelines where models generate their own training data: content-based inspection cannot detect such transmission, and even preference-contradicting content fails to prevent it.