Convolutional Neural Network Simplification with Progressive Retraining

TL;DR

This paper introduces a novel CNN simplification method called progressive retraining, which selectively retrains layers after kernel elimination based on relevance criteria, leading to more effective model simplification and better performance than existing methods.

Contribution

The paper proposes a new layer-by-layer kernel elimination approach with progressive retraining guided by relevance criteria, improving CNN simplification effectiveness.

Findings

Outperforms two popular kernel pruning methods and a state-of-the-art approach.

Achieves significant model simplification while maintaining accuracy.

Demonstrates effectiveness on four challenging image datasets.

Abstract

Kernel pruning methods have been proposed to speed up, simplify, and improve explanation of convolutional neural network (CNN) models. However, the effectiveness of a simplified model is often below the original one. In this letter, we present new methods based on objective and subjective relevance criteria for kernel elimination in a layer-by-layer fashion. During the process, a CNN model is retrained only when the current layer is entirely simplified, by adjusting the weights from the next layer to the first one and preserving weights of subsequent layers not involved in the process. We call this strategy \emph{progressive retraining}, differently from kernel pruning methods that usually retrain the entire model after each simplification action -- e.g., the elimination of one or a few kernels. Our subjective relevance criterion exploits the ability of humans in recognizing visual patterns and improves the designer's understanding of the simplification process. The combination of suitable relevance criteria and progressive retraining shows that our methods can increase effectiveness with considerable model simplification. We also demonstrate that our methods can provide better results than two popular ones and another one from the state-of-the-art using four challenging image datasets.