The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition

TL;DR

This paper uncovers a vulnerability in tokenizer transplant used in model composition, demonstrating a stealthy attack that sabotages language model outputs without detection, posing risks to modular AI systems.

Contribution

It introduces a novel, training-free attack exploiting tokenizer transplant geometry, revealing a hidden security risk in model composition workflows.

Findings

Attack is training-free and evades detection

Sabotages model outputs while maintaining donor utility

Persists against fine-tuning and weight merging

Abstract

The open-weight language model ecosystem is increasingly defined by model composition techniques (such as weight merging, speculative decoding, and vocabulary expansion) that remix capabilities from diverse sources. A critical prerequisite for applying these methods across different model families is tokenizer transplant, which aligns incompatible vocabularies to a shared embedding space. We demonstrate that this essential interoperability step introduces a supply-chain vulnerability: we engineer a single breaker token that is functionally inert in a donor model yet reliably reconstructs into a high-salience malicious feature after transplant into a base model. By exploiting the geometry of coefficient reuse, our attack sabotages the base model's generation while leaving the donor's utility statistically indistinguishable from nominal behavior. We formalize this as a dual-objective optimization problem and instantiate the attack using a sparse solver. Empirically, the attack is training-free and evades outlier detection, while demonstrating structural persistence against fine-tuning and weight merging, highlighting a hidden risk in the pipeline of modular AI composition. Code is available at https://github.com/xz-liu/tokenforge