A scalable convolutional neural network for task-specified scenarios via knowledge distillation

TL;DR

This paper introduces a knowledge distillation method to create simplified, task-specific convolutional neural networks that reduce redundancy and meet resource constraints, demonstrated on MNIST and CIFAR10 datasets.

Contribution

It proposes a novel task-specified knowledge distillation algorithm to derive efficient CNN models with controlled computation and minimal accuracy loss.

Findings

Redundancy in CNNs scales with task complexity.

Task-specific redundancy can be effectively removed.

The approach maintains accuracy while reducing model size.

Abstract

In this paper, we explore the redundancy in convolutional neural network, which scales with the complexity of vision tasks. Considering that many front-end visual systems are interested in only a limited range of visual targets, the removing of task-specified network redundancy can promote a wide range of potential applications. We propose a task-specified knowledge distillation algorithm to derive a simplified model with pre-set computation cost and minimized accuracy loss, which suits the resource constraint front-end systems well. Experiments on the MNIST and CIFAR10 datasets demonstrate the feasibility of the proposed approach as well as the existence of task-specified redundancy.