StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

TL;DR

StatsMerging is a novel, lightweight model merging method that uses weight distribution statistics and task-specific teacher distillation to effectively combine models without ground truth labels, improving accuracy and robustness.

Contribution

It introduces a new merging approach leveraging SVD-based weight importance, a lightweight learner for distribution modeling, and task-specific teacher distillation for heterogeneous models.

Findings

Outperforms state-of-the-art methods in accuracy

Demonstrates strong generalization to unseen tasks

Shows robustness to image quality variations

Abstract

Model merging has emerged as a promising solution to accommodate multiple large models within constrained memory budgets. We present StatsMerging, a novel lightweight learning-based model merging method guided by weight distribution statistics without requiring ground truth labels or test samples. StatsMerging offers three key advantages: (1) It uniquely leverages singular values from singular value decomposition (SVD) to capture task-specific weight distributions, serving as a proxy for task importance to guide task coefficient prediction; (2) It employs a lightweight learner StatsMergeLearner to model the weight distributions of task-specific pre-trained models, improving generalization and enhancing adaptation to unseen samples; (3) It introduces Task-Specific Teacher Distillation for merging vision models with heterogeneous architectures, a merging learning paradigm that avoids costly ground-truth labels by task-specific teacher distillation. Notably, we present two types of knowledge distillation, (a) distilling knowledge from task-specific models to StatsMergeLearner; and (b) distilling knowledge from models with heterogeneous architectures prior to merging. Extensive experiments across eight tasks demonstrate the effectiveness of StatsMerging. Our results show that StatsMerging outperforms state-of-the-art techniques in terms of overall accuracy, generalization to unseen tasks, and robustness to image quality variations.