When Safe Models Merge into Danger: Exploiting Latent Vulnerabilities in LLM Fusion

TL;DR

This paper uncovers security vulnerabilities in model merging of LLMs, demonstrating how malicious components can be embedded to produce unsafe merged models while source models stay safe.

Contribution

It introduces TrojanMerge, a novel attack framework that systematically embeds latent malicious components into models, exposing security risks in current merging practices.

Findings

TrojanMerge achieves high harmful response rates in merged models.

Source models maintain safety scores similar to unmodified models.

The attack is effective across multiple merging algorithms and hyperparameters.

Abstract

Model merging has emerged as a powerful technique for combining specialized capabilities from multiple fine-tuned LLMs without additional training costs. However, the security implications of this widely-adopted practice remain critically underexplored. In this work, we reveal that model merging introduces a novel attack surface that can be systematically exploited to compromise safety alignment. We present TrojanMerge,, a framework that embeds latent malicious components into source models that remain individually benign but produce severely misaligned models when merged. Our key insight is formulating this attack as a constrained optimization problem: we construct perturbations that preserve source model safety through directional consistency constraints, maintain capabilities via Frobenius directional alignment constraints, yet combine during merging to form pre-computed attack vectors. Extensive experiments across 9 LLMs from 3 model families demonstrate that TrojanMerge, consistently achieves high harmful response rates in merged models while source models maintain safety scores comparable to unmodified versions. Our attack succeeds across diverse merging algorithms and remains effective under various hyperparameter configurations. These findings expose fundamental vulnerabilities in current model merging practices and highlight the urgent need for security-aware mechanisms.