Geodesic Mode Connectivity
Charlie Tan, Theodore Long, Sarah Zhao, Rudolf Laine

TL;DR
This paper explores the concept of mode connectivity in neural networks through the lens of Information Geometry, proposing geodesic paths as a means to connect different trained models with low loss.
Contribution
It introduces a novel geometric perspective by framing mode connectivity as geodesics in the space of neural network distributions, along with an algorithm to approximate these paths.
Findings
Geodesic paths effectively connect trained models with low loss.
The proposed algorithm successfully approximates geodesics in the parameter space.
Geometric interpretation enhances understanding of neural network loss landscapes.
Abstract
Mode connectivity is a phenomenon where trained models are connected by a path of low loss. We reframe this in the context of Information Geometry, where neural networks are studied as spaces of parameterized distributions with curved geometry. We hypothesize that shortest paths in these spaces, known as geodesics, correspond to mode-connecting paths in the loss landscape. We propose an algorithm to approximate geodesics and demonstrate that they achieve mode connectivity.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Model Reduction and Neural Networks · Computational Physics and Python Applications
