Superpose Task-specific Features for Model Merging

TL;DR

This paper introduces a novel model merging method that superposes task-specific features based on linear representations, effectively preserving multi-task capabilities without additional training.

Contribution

It proposes a linear system-based approach to merge task-specific features, improving multi-task performance over existing methods.

Findings

Outperforms existing model merging techniques on diverse benchmarks

Effectively preserves multi-task capabilities in merged models

Demonstrates robustness across different neural network architectures

Abstract

Model merging enables powerful capabilities in neural networks without requiring additional training. In this paper, we introduce a novel perspective on model merging by leveraging the fundamental mechanisms of neural network representation. Our approach is motivated by the linear representation hypothesis, which states that neural networks encode information through linear combinations of feature vectors. We propose a method that superposes task-specific features from individual models into a merged model. Our approach specifically targets linear transformation matrices, which are crucial for feature activation and extraction in deep networks. By formulating the merging process as a linear system, we can preserve task-specific features from individual models and create merged models that effectively maintain multi-task capabilities compared to existing methods. Extensive experiments across diverse benchmarks and models demonstrate that our method outperforms existing techniques. Code is available at https://github.com/LARS-research/STF.