NestedNet: Learning Nested Sparse Structures in Deep Neural Networks

TL;DR

NestedNet introduces a multi-level sparse neural network architecture that shares parameters across levels, enabling resource adaptability and multi-task learning within a single model, improving efficiency and versatility.

Contribution

It proposes a novel nested sparse network framework with shared parameters, allowing resource-aware adaptation and multi-task learning in deep neural networks.

Findings

Performs competitively in compression and distillation tasks

Achieves resource adaptability for diverse device constraints

Enables multi-task learning within a single network

Abstract

Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep architecture for multiple devices with different resources. When a new device or circumstantial condition requires a new deep architecture, it is necessary to construct and train a new network from scratch. In this work, we propose a novel deep learning framework, called a nested sparse network, which exploits an n-in-1-type nested structure in a neural network. A nested sparse network consists of multiple levels of networks with a different sparsity ratio associated with each level, and higher level networks share parameters with lower level networks to enable stable nested learning. The proposed framework realizes a resource-aware versatile architecture as the same network can meet diverse resource requirements. Moreover, the proposed nested network can learn different forms of knowledge in its internal networks at different levels, enabling multiple tasks using a single network, such as coarse-to-fine hierarchical classification. In order to train the proposed nested sparse network, we propose efficient weight connection learning and channel and layer scheduling strategies. We evaluate our network in multiple tasks, including adaptive deep compression, knowledge distillation, and learning class hierarchy, and demonstrate that nested sparse networks perform competitively, but more efficiently, compared to existing methods.