Connecting Independently Trained Modes via Layer-Wise Connectivity

TL;DR

This paper introduces a new empirical algorithm that effectively connects independently trained neural network models across diverse modern architectures, enhancing the understanding of mode connectivity in complex models.

Contribution

The authors propose a generalized empirical method for mode connectivity that works across various modern architectures and training conditions, surpassing previous limitations.

Findings

Supports a wider range of architectures including MobileNet, ShuffleNet, and EfficientNet.

Provides more consistent connectivity paths between independently trained models.

Enables connecting modes trained with different hyperparameters.

Abstract

Empirical and theoretical studies have shown that continuous low-loss paths can be constructed between independently trained neural network models. This phenomenon, known as mode connectivity, refers to the existence of such paths between distinct modes-i.e., well-trained solutions in parameter space. However, existing empirical methods are primarily effective for older and relatively simple architectures such as basic CNNs, VGG, and ResNet, raising concerns about their applicability to modern and structurally diverse models. In this work, we propose a new empirical algorithm for connecting independently trained modes that generalizes beyond traditional architectures and supports a broader range of networks, including MobileNet, ShuffleNet, EfficientNet, RegNet, Deep Layer Aggregation (DLA), and Compact Convolutional Transformers (CCT). In addition to broader applicability, the proposed method yields more consistent connectivity paths across independently trained mode pairs and supports connecting modes obtained with different training hyperparameters.