Sparsifying networks by traversing Geodesics

TL;DR

This paper introduces a geometric framework to identify high-performance paths in neural network weight spaces, enabling effective sparsification and potentially addressing issues like catastrophic forgetting.

Contribution

It proposes a novel mathematical approach to evaluate geodesics in functional space, facilitating the transition from dense to sparse networks while maintaining performance.

Findings

Successful application on VGG-11 with CIFAR-10

Effective sparsification of MLPs on MNIST

Framework's versatility for various problems

Abstract

The geometry of weight spaces and functional manifolds of neural networks play an important role towards 'understanding' the intricacies of ML. In this paper, we attempt to solve certain open questions in ML, by viewing them through the lens of geometry, ultimately relating it to the discovery of points or paths of equivalent function in these spaces. We propose a mathematical framework to evaluate geodesics in the functional space, to find high-performance paths from a dense network to its sparser counterpart. Our results are obtained on VGG-11 trained on CIFAR-10 and MLP's trained on MNIST. Broadly, we demonstrate that the framework is general, and can be applied to a wide variety of problems, ranging from sparsification to alleviating catastrophic forgetting.