Distilling Critical Paths in Convolutional Neural Networks

TL;DR

This paper analyzes the internal workings of convolutional neural networks to identify critical information pathways and introduces a distillation method that customizes and compresses models for resource-constrained deployment.

Contribution

It reveals class-specific critical paths in CNNs and proposes a distillation technique to create smaller, efficient models tailored to specific tasks.

Findings

Critical paths vary across classes and are highly task-specific.

The proposed distillation method significantly reduces model size and computation.

Customized models maintain high accuracy on target tasks.

Abstract

Neural network compression and acceleration are widely demanded currently due to the resource constraints on most deployment targets. In this paper, through analyzing the filter activation, gradients, and visualizing the filters' functionality in convolutional neural networks, we show that the filters in higher layers learn extremely task-specific features, which are exclusive for only a small subset of the overall tasks, or even a single class. Based on such findings, we reveal the critical paths of information flow for different classes. And by their intrinsic property of exclusiveness, we propose a critical path distillation method, which can effectively customize the convolutional neural networks to small ones with much smaller model size and less computation.