On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks

TL;DR

This paper introduces CLAFusion, a novel framework for fusing neural networks with different layer counts using cross-layer alignment, enhancing model accuracy and efficiency in various applications.

Contribution

We propose CLAFusion, a new method for fusing heterogeneous neural networks with different layer numbers via cross-layer alignment and dynamic programming.

Findings

CLAFusion improves residual network accuracy on CIFAR and Tiny-ImageNet datasets.

The method effectively balances layers before fusion, enabling better model compression.

CLAFusion benefits knowledge distillation in teacher-student setups.

Abstract

Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting.