Distill-Net: Application-Specific Distillation of Deep Convolutional Neural Networks for Resource-Constrained IoT Platforms

TL;DR

This paper introduces Distill-Net, a method to simplify deep CNNs for IoT devices by focusing on relevant classes, enabling fast and accurate inference within resource constraints.

Contribution

It proposes a novel distillation approach that creates application-specific CNN cores optimized for resource-limited IoT platforms.

Findings

Achieves high accuracy on target classes with reduced model size.

Improves inference speed on resource-constrained IoT devices.

Balances accuracy, efficiency, and development productivity.

Abstract

Many Internet-of-Things (IoT) applications demand fast and accurate understanding of a few key events in their surrounding environment. Deep Convolutional Neural Networks (CNNs) have emerged as an effective approach to understand speech, images, and similar high dimensional data types. Algorithmic performance of modern CNNs, however, fundamentally relies on learning class-agnostic hierarchical features that only exist in comprehensive training datasets with many classes. As a result, fast inference using CNNs trained on such datasets is prohibitive for most resource-constrained IoT platforms. To bridge this gap, we present a principled and practical methodology for distilling a complex modern CNN that is trained to effectively recognize many different classes of input data into an application-dependent essential core that not only recognizes the few classes of interest to the application accurately, but also runs efficiently on platforms with limited resources. Experimental results confirm that our approach strikes a favorable balance between classification accuracy (application constraint), inference efficiency (platform constraint), and productive development of new applications (business constraint).