Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations

TL;DR

This paper introduces a novel training algorithm that finds low-rank subnetworks within neural networks by restricting weight matrices to low-rank manifolds, significantly reducing training and evaluation resources.

Contribution

It develops a dynamic low-rank training method using matrix differential equations, enabling automatic rank adaptation and providing theoretical guarantees.

Findings

Reduces training and evaluation time and memory usage.

Automatically adapts low-rank structures during training.

Demonstrates effectiveness on various neural network architectures.

Abstract

Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. Remarkably, these subnetworks are determined and adapted already during the training phase and the overall time and memory resources required by both training and evaluating them are significantly reduced. The main idea is to restrict the weight matrices to a low-rank manifold and to update the low-rank factors rather than the full matrix during training. To derive training updates that are restricted to the prescribed manifold, we employ techniques from dynamic model order reduction for matrix differential equations. This allows us to provide approximation, stability, and descent guarantees. Moreover, our method automatically and dynamically adapts the ranks during training to achieve the desired approximation accuracy. The efficiency of the proposed method is demonstrated through a variety of numerical experiments on fully-connected and convolutional networks.