DRESS: Dynamic REal-time Sparse Subnets

TL;DR

DRESS introduces a novel training algorithm for dynamically adapting sparse sub-networks within a backbone neural network, enabling efficient on-device resource management and improved accuracy on vision tasks.

Contribution

It proposes a new training method that jointly optimizes multiple sparse sub-networks via row-based unstructured sparsity, reducing re-configuration overhead and enhancing performance.

Findings

DRESS achieves higher accuracy than existing sub-network methods.

It enables efficient on-device adaptation with minimal storage overhead.

Extensive experiments validate its effectiveness on vision datasets.

Abstract

The limited and dynamically varied resources on edge devices motivate us to deploy an optimized deep neural network that can adapt its sub-networks to fit in different resource constraints. However, existing works often build sub-networks through searching different network architectures in a hand-crafted sampling space, which not only can result in a subpar performance but also may cause on-device re-configuration overhead. In this paper, we propose a novel training algorithm, Dynamic REal-time Sparse Subnets (DRESS). DRESS samples multiple sub-networks from the same backbone network through row-based unstructured sparsity, and jointly trains these sub-networks in parallel with weighted loss. DRESS also exploits strategies including parameter reusing and row-based fine-grained sampling for efficient storage consumption and efficient on-device adaptation. Extensive experiments on public vision datasets show that DRESS yields significantly higher accuracy than state-of-the-art sub-networks.