Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization

TL;DR

This paper introduces SIFT, a spectral optimization framework that enables controllable model steering by orthogonalizing subspaces, effectively addressing interference between objectives and constraints in model training.

Contribution

The paper presents a novel spectral interference-free training method, SIFT, that improves constrained model adaptation across diverse applications by mitigating objective-constraint conflicts.

Findings

SIFT outperforms control-based and control-free baselines in multiple tasks.

Spectral orthogonalization resolves cross-task interference during training.

SIFT achieves robust performance improvements in safety, unlearning, speech, and hallucination mitigation.

Abstract

Foundation models, such as large language models (LLMs), are powerful but often require customization before deployment to satisfy practical constraints such as safety, privacy, and task-specific requirements, leading to "constrained" optimization problems for model steering and adaptation. However, solving such problems remains largely underexplored and is particularly challenging due to interference between the primary objective and constraint objectives during optimization. In this paper, we propose a subspace control framework for constrained model training. Specifically, (i) we first analyze, from a model merging perspective, how spectral cross-task interference arises and show that it can be resolved via a one-shot solution that orthogonalizes the merged subspace; (ii) we establish a connection between this solution and gradient orthogonalization in the spectral optimizer Muon; and (iii) building on these insights, we introduce SIFT (spectral interference-free training), which leverages a localization scheme to selectively intervene during optimization, enabling controllable updates that mitigate objective-constraint conflicts. We evaluate SIFT across four representative applications: (a) machine unlearning, (b) safety alignment, (c) text-to-speech adaptation, and (d) hallucination mitigation. Compared to both control-based and control-free baselines, SIFT consistently achieves substantial and robust performance improvements across all tasks. Code is available at https://github.com/OPTML-Group/SIFT.