SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search

TL;DR

SVD-NAS introduces a novel framework combining low-rank approximation and neural architecture search to efficiently compress neural networks, achieving higher accuracy and better resource reduction than existing methods.

Contribution

It proposes a new Low-Rank architecture space and a gradient-based search method, enabling more effective exploration and compression of neural networks.

Findings

Achieves 2.06-12.85pp higher accuracy on ImageNet.

Reduces parameters, FLOPS, and latency significantly.

Outperforms state-of-the-art methods in data-limited settings.

Abstract

The task of compressing pre-trained Deep Neural Networks has attracted wide interest of the research community due to its great benefits in freeing practitioners from data access requirements. In this domain, low-rank approximation is a promising method, but existing solutions considered a restricted number of design choices and failed to efficiently explore the design space, which lead to severe accuracy degradation and limited compression ratio achieved. To address the above limitations, this work proposes the SVD-NAS framework that couples the domains of low-rank approximation and neural architecture search. SVD-NAS generalises and expands the design choices of previous works by introducing the Low-Rank architecture space, LR-space, which is a more fine-grained design space of low-rank approximation. Afterwards, this work proposes a gradient-descent-based search for efficiently traversing the LR-space. This finer and more thorough exploration of the possible design choices results in improved accuracy as well as reduction in parameters, FLOPS, and latency of a CNN model. Results demonstrate that the SVD-NAS achieves 2.06-12.85pp higher accuracy on ImageNet than state-of-the-art methods under the data-limited problem setting. SVD-NAS is open-sourced at https://github.com/Yu-Zhewen/SVD-NAS.