A Progressive Sub-Network Searching Framework for Dynamic Inference

TL;DR

This paper introduces a progressive sub-network searching framework that enhances the dynamic inference capability of deep neural networks, allowing better accuracy-complexity trade-offs post-deployment through efficient sub-net sampling and selection techniques.

Contribution

It proposes a novel progressive sub-net searching framework with techniques like trainable noise ranking and sub-net re-selection to improve dynamic inference performance.

Findings

Outperforms prior methods on CIFAR-10 and ImageNet datasets.

Achieves 4.4% better accuracy in dynamic inference over Universally-Slimmable-Network.

Demonstrates effectiveness across different network structures.

Abstract

Many techniques have been developed, such as model compression, to make Deep Neural Networks (DNNs) inference more efficiently. Nevertheless, DNNs still lack excellent run-time dynamic inference capability to enable users trade-off accuracy and computation complexity (i.e., latency on target hardware) after model deployment, based on dynamic requirements and environments. Such research direction recently draws great attention, where one realization is to train the target DNN through a multiple-term objective function, which consists of cross-entropy terms from multiple sub-nets. Our investigation in this work show that the performance of dynamic inference highly relies on the quality of sub-net sampling. With objective to construct a dynamic DNN and search multiple high quality sub-nets with minimal searching cost, we propose a progressive sub-net searching framework, which is embedded with several effective techniques, including trainable noise ranking, channel group and fine-tuning threshold setting, sub-nets re-selection. The proposed framework empowers the target DNN with better dynamic inference capability, which outperforms prior works on both CIFAR-10 and ImageNet dataset via comprehensive experiments on different network structures. Taken ResNet18 as an example, our proposed method achieves much better dynamic inference accuracy compared with prior popular Universally-Slimmable-Network by 4.4%-maximally and 2.3%-averagely in ImageNet dataset with the same model size.