NetCut: Real-Time DNN Inference Using Layer Removal

TL;DR

This paper introduces NetCut, a method for constructing and selecting trimmed neural networks through layer removal to meet real-time inference deadlines while improving accuracy and reducing exploration time.

Contribution

It proposes layer removal for creating transfer learning-based trimmed networks and a methodology for efficiently selecting networks that meet specific latency constraints.

Findings

TRNs can expand the latency-accuracy Pareto frontier.

NetCut achieves up to 10.43% accuracy improvement under deadlines.

27x reduction in network exploration time.

Abstract

Deep Learning plays a significant role in assisting humans in many aspects of their lives. As these networks tend to get deeper over time, they extract more features to increase accuracy at the cost of additional inference latency. This accuracy-performance trade-off makes it more challenging for Embedded Systems, as resource-constrained processors with strict deadlines, to deploy them efficiently. This can lead to selection of networks that can prematurely meet a specified deadline with excess slack time that could have potentially contributed to increased accuracy. In this work, we propose: (i) the concept of layer removal as a means of constructing TRimmed Networks (TRNs) that are based on removing problem-specific features of a pretrained network used in transfer learning, and (ii) NetCut, a methodology based on an empirical or an analytical latency estimator, which only proposes and retrains TRNs that can meet the application's deadline, hence reducing the exploration time significantly. We demonstrate that TRNs can expand the Pareto frontier that trades off latency and accuracy to provide networks that can meet arbitrary deadlines with potential accuracy improvement over off-the-shelf networks. Our experimental results show that such utilization of TRNs, while transferring to a simpler dataset, in combination with NetCut, can lead to the proposal of networks that can achieve relative accuracy improvement of up to 10.43% among existing off-the-shelf neural architectures while meeting a specific deadline, and 27x speedup in exploration time.