Landscaping Linear Mode Connectivity
Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard, Sch\"olkopf, Thomas Hofmann

TL;DR
This paper investigates the conditions under which linear mode connectivity occurs in neural network loss landscapes, proposing a topographical model and analyzing barrier heights to better understand network solution connectivity.
Contribution
It introduces a 'mountainside and ridge' topographical model of the loss landscape and provides a theoretical analysis of barrier heights related to linear mode connectivity.
Findings
A topographical model explains LMC occurrence.
Barrier height predicts layer-wise LMC.
Empirical support for the barrier analysis.
Abstract
The presence of linear paths in parameter space between two different network solutions in certain cases, i.e., linear mode connectivity (LMC), has garnered interest from both theoretical and practical fronts. There has been significant research that either practically designs algorithms catered for connecting networks by adjusting for the permutation symmetries as well as some others that more theoretically construct paths through which networks can be connected. Yet, the core reasons for the occurrence of LMC, when in fact it does occur, in the highly non-convex loss landscapes of neural networks are far from clear. In this work, we take a step towards understanding it by providing a model of how the loss landscape needs to behave topographically for LMC (or the lack thereof) to manifest. Concretely, we present a `mountainside and ridge' perspective that helps to neatly tie together…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEnergy Efficient Wireless Sensor Networks
