From Parameter to Representation: A Closed-Form Approach for Controllable Model Merging

TL;DR

This paper introduces a novel closed-form method for controllable model merging that efficiently generates Pareto-optimal models by directly correcting representations, significantly reducing computational costs compared to previous iterative approaches.

Contribution

The authors propose a representation correction approach using a closed-form solution, replacing costly offline optimization with a single, scalable computation for multi-task model merging.

Findings

Produces a superior Pareto front with better preference alignment

Reduces computational complexity from exponential to linear in the number of tasks

Enables on-the-fly, preference-aware model generation

Abstract

Model merging combines expert models for multitask performance but faces challenges from parameter interference. This has sparked recent interest in controllable model merging, giving users the ability to explicitly balance performance trade-offs. Existing approaches employ a compile-then-query paradigm, performing a costly offline multi-objective optimization to enable fast, preference-aware model generation. This offline stage typically involves iterative search or dedicated training, with complexity that grows exponentially with the number of tasks. To overcome these limitations, we shift the perspective from parameter-space optimization to a direct correction of the model's final representation. Our approach models this correction as an optimal linear transformation, yielding a closed-form solution that replaces the entire offline optimization process with a single-step, architecture-agnostic computation. This solution directly incorporates user preferences, allowing a Pareto-optimal model to be generated on-the-fly with complexity that scales linearly with the number of tasks. Experimental results show our method generates a superior Pareto front with more precise preference alignment and drastically reduced computational cost.