Dynamic Slimmable Network
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li and, Xiaojun Chang

TL;DR
The paper introduces Dynamic Slimmable Network (DS-Net), a hardware-efficient dynamic network that adjusts filter numbers at test time using a novel double-headed gate, achieving significant computation reduction and acceleration with minimal accuracy loss.
Contribution
The paper proposes DS-Net with a double-headed dynamic gate and a two-stage training scheme, enabling efficient dynamic inference and improved supernet training for network slimming.
Findings
Achieves 2-4x computation reduction on ImageNet.
Real-world acceleration of 1.62x over ResNet-50 and MobileNet.
Outperforms static and dynamic compression methods by up to 5.9%.
Abstract
Current dynamic networks and dynamic pruning methods have shown their promising capability in reducing theoretical computation complexity. However, dynamic sparse patterns on convolutional filters fail to achieve actual acceleration in real-world implementation, due to the extra burden of indexing, weight-copying, or zero-masking. Here, we explore a dynamic network slimming regime, named Dynamic Slimmable Network (DS-Net), which aims to achieve good hardware-efficiency via dynamically adjusting filter numbers of networks at test time with respect to different inputs, while keeping filters stored statically and contiguously in hardware to prevent the extra burden. Our DS-Net is empowered with the ability of dynamic inference by the proposed double-headed dynamic gate that comprises an attention head and a slimming head to predictively adjust network width with negligible extra…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning
MethodsPruning
