A correlation-permutation approach for speech-music encoders model merging

TL;DR

This paper introduces a correlation-permutation method to align and merge speech and music encoders, enabling the creation of a unified audio model with improved performance and minimal retraining.

Contribution

It extends the Git Re-Basin approach to transformer layers, allowing effective merging of independently trained speech and music encoders through correlation-based layer alignment.

Findings

Merged model retains speech capabilities.

Achieves 14.83 point improvement over linear interpolation.

Enables creation of unified audio models from separate encoders.

Abstract

Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders.