FeatCal: Feature Calibration for Post-Merging Models

TL;DR

FeatCal is a novel feature calibration method for post-merging models that reduces feature drift and improves performance efficiently without iterative training, demonstrated on CLIP and GLUE benchmarks.

Contribution

It introduces a layer-by-layer weight calibration approach using a small set, with a closed-form solution, outperforming existing baselines in efficiency and accuracy.

Findings

FeatCal achieves higher accuracy than baselines on CLIP and GLUE benchmarks.

It requires fewer examples and less time for calibration, demonstrating better sample efficiency.

The method effectively reduces feature drift and output drift in merged models.

Abstract

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.