Language-Switching Triggers Take a Latent Detour Through Language Models

TL;DR

This paper uncovers the internal circuit mechanism behind a language-switching backdoor in a large language model, revealing how a Latin trigger redirects output from English to French through a three-phase process.

Contribution

It identifies and decomposes the circuit responsible for a language-switching backdoor in a language model, highlighting the latent encoding and bottleneck structure involved.

Findings

The backdoor trigger operates through a three-phase circuit in the model.

The trigger signal propagates orthogonally to the natural language-identity direction.

Disrupting the bottleneck position effectively mitigates the backdoor.

Abstract

Backdoor attacks on language models pose a growing security concern, yet the internal mechanisms by which a trigger sequence hijacks model computations remain poorly understood. We identify a circuit underlying a language-switching backdoor in an 8B-parameter autoregressive language model, where a three-word Latin trigger (nine tokens) redirects English output to French. We decompose the circuit into three phases: (1) distributed attention heads at early layers compose the trigger tokens into the last sequence position; (2) the resulting signal propagates through mid-layers in a subspace orthogonal to the model's natural language-identity direction; (3) the MLP at the final layer converts this latent signal into French logits. The entire circuit flows through a serial bottleneck at a single position: corrupting that position at any layer entirely mitigate the trigger but also hinder the model's capabilities. The orthogonal latent encoding suggests that defenses that search for language-like signals in intermediate representations would miss this trigger entirely.