Mode Combinability: Exploring Convex Combinations of Permutation Aligned Models

TL;DR

This paper investigates the concept of mode combinability in neural networks, showing that convex combinations of permutation-aligned models often result in low-loss models, extending the idea of linear mode connectivity.

Contribution

It introduces the notion of mode combinability, demonstrating that convex combinations of permutation-aligned models can maintain low loss and revealing properties like transitivity and robustness.

Findings

Convex combinations of models often lie on low-loss surfaces.

Mode combinability extends linear mode connectivity.

Model combinations retain functional differences despite perturbations.

Abstract

We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors $\Theta_A$ and $\Theta_B$ of size $d$. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube $[0,1]^{d}$ and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models.