PRANC: Pseudo RAndom Networks for Compacting deep models

TL;DR

PRANC introduces a method to reparametrize deep models as linear combinations of random, frozen networks, enabling significant compression and efficient storage, especially useful for edge devices and federated learning.

Contribution

This work presents PRANC, a novel framework that compresses deep models by representing them as linear combinations of random basis networks, facilitating efficient storage and communication.

Findings

PRANC compresses models nearly 100 times more effectively than baselines.

PRANC enables memory-efficient inference by generating weights on the fly.

PRANC outperforms existing methods in image classification tasks.

Abstract

We demonstrate that a deep model can be reparametrized as a linear combination of several randomly initialized and frozen deep models in the weight space. During training, we seek local minima that reside within the subspace spanned by these random models (i.e., `basis' networks). Our framework, PRANC, enables significant compaction of a deep model. The model can be reconstructed using a single scalar `seed,' employed to generate the pseudo-random `basis' networks, together with the learned linear mixture coefficients. In practical applications, PRANC addresses the challenge of efficiently storing and communicating deep models, a common bottleneck in several scenarios, including multi-agent learning, continual learners, federated systems, and edge devices, among others. In this study, we employ PRANC to condense image classification models and compress images by compacting their associated implicit neural networks. PRANC outperforms baselines with a large margin on image classification when compressing a deep model almost $100$ times. Moreover, we show that PRANC enables memory-efficient inference by generating layer-wise weights on the fly. The source code of PRANC is here: \url{https://github.com/UCDvision/PRANC}