Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents

TL;DR

Fine-tuning vision-language models on narrow harmful datasets causes significant and broad misalignment, which is difficult to fully mitigate, raising concerns about safety in continual learning scenarios.

Contribution

This paper demonstrates that narrow-domain fine-tuning induces broad misalignment in vision-language models and explores mitigation strategies, highlighting challenges in maintaining safety alignment.

Findings

Misalignment scales with LoRA rank and is higher in multimodal evaluation.

Even 10 ext{ } of harmful data causes significant degradation.

Harmful behaviors are low-dimensional, captured in 10 principal components.

Abstract

Lifelong multimodal agents must continuously adapt to new tasks through post-training, but this creates a fundamental tension between acquiring capabilities and preserving safety alignment. We demonstrate that fine-tuning aligned vision-language models on narrow-domain harmful datasets induces severe emergent misalignment that generalizes broadly across unrelated tasks and modalities. Through experiments on Gemma3-4B, we show that misalignment scales monotonically with LoRA rank, and that multimodal evaluation reveals substantially higher misalignment ($70.71 \pm 1.22$ at $r=128$) than text-only evaluation ($41.19 \pm 2.51$), suggesting that unimodal safety benchmarks may underestimate alignment degradation in vision-language models. Critically, even 10\% harmful data in the training mixture induces substantial alignment degradation. Geometric analysis reveals that harmful behaviors occupy a remarkably low-dimensional subspace, with the majority of misalignment information captured in 10 principal components. To mitigate misalignment, we evaluate two strategies: benign narrow fine-tuning and activation-based steering. While both approaches substantially reduce misalignment, neither completely removes the learned harmful behaviors. Our findings highlight the need for robust continual learning frameworks, as current post-training paradigms may not sufficiently preserve alignment in post-deployment settings.