Feature Flow Regularization: Improving Structured Sparsity in Deep Neural Networks

TL;DR

This paper introduces feature flow regularization (FFR), a novel method that enhances structured sparsity in deep neural networks by controlling feature evolution, leading to more effective filter pruning without complex procedures.

Contribution

The proposed FFR method is a simple regularization strategy that improves structured sparsity and pruning efficiency in DNNs by regulating feature flow evolution.

Findings

FFR significantly improves sparsity in DNNs.

Pruning results are comparable or superior to state-of-the-art methods.

Effective across various architectures and datasets.

Abstract

Pruning is a model compression method that removes redundant parameters in deep neural networks (DNNs) while maintaining accuracy. Most available filter pruning methods require complex treatments such as iterative pruning, features statistics/ranking, or additional optimization designs in the training process. In this paper, we propose a simple and effective regularization strategy from a new perspective of evolution of features, which we call feature flow regularization (FFR), for improving structured sparsity and filter pruning in DNNs. Specifically, FFR imposes controls on the gradient and curvature of feature flow along the neural network, which implicitly increases the sparsity of the parameters. The principle behind FFR is that coherent and smooth evolution of features will lead to an efficient network that avoids redundant parameters. The high structured sparsity obtained from FFR enables us to prune filters effectively. Experiments with VGGNets, ResNets on CIFAR-10/100, and Tiny ImageNet datasets demonstrate that FFR can significantly improve both unstructured and structured sparsity. Our pruning results in terms of reduction of parameters and FLOPs are comparable to or even better than those of state-of-the-art pruning methods.