Transport and Merge: Cross-Architecture Merging for Large Language Models

TL;DR

This paper introduces a novel cross-architecture merging framework using optimal transport to transfer knowledge from large, high-resource language models to smaller, low-resource models, improving their performance.

Contribution

The work presents the first cross-architecture merging method based on optimal transport, enabling effective knowledge transfer between heterogeneous models.

Findings

Consistent performance improvements on low-resource languages.

Effective transfer with limited input data.

Outperforms existing model merging approaches.

Abstract

Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.