Generalized Linear Mode Connectivity for Transformers

TL;DR

This paper introduces a unified symmetry framework for analyzing the loss landscapes of Transformers, revealing low-loss paths between independently trained models and across different architectures, thus deepening understanding of model space geometry.

Contribution

It broadens the scope of symmetry considerations in neural networks to include multiple classes, enabling discovery of linear paths between models and across architectures, which was not possible before.

Findings

Low- and zero-barrier linear paths between trained Transformers.

Framework extends to multi-model and heterogeneous architectures.

Reveals deeper structure in the loss landscape.

Abstract

Understanding the geometry of neural network loss landscapes is a central question in deep learning, with implications for generalization and optimization. A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths despite appearing to lie in separate loss basins. However, this is often obscured by symmetries in parameter space -- such as neuron permutations -- which make functionally equivalent models appear dissimilar. Prior work has predominantly focused on neuron reordering through permutations, but such approaches are limited in scope and fail to capture the richer symmetries exhibited by modern architectures such as Transformers. In this work, we introduce a unified framework that captures four symmetry classes -- permutations, semi-permutations, orthogonal transformations, and general invertible maps -- broadening the set of valid reparameterizations and subsuming many previous approaches as special cases. Crucially, this generalization enables, for the first time, the discovery of low- and zero-barrier linear interpolation paths between independently trained Vision Transformers and GPT-2 models. Furthermore, our framework extends beyond pairwise alignment to multi-model and width-heterogeneous settings, enabling alignment across architectures of different sizes. These results reveal deeper structure in the loss landscape and underscore the importance of symmetry-aware analysis for understanding model space geometry.