FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees

TL;DR

This paper introduces FeTa, a fast DNN pruning algorithm based on difference of convex functions that maintains accuracy with minimal retraining and provides theoretical guarantees on generalization error growth.

Contribution

Proposes a novel, efficient DNN pruning algorithm with theoretical generalization error guarantees, reducing retraining needs and computational cost.

Findings

FeTa is significantly faster than existing pruning methods.

Theoretical analysis links pruning to bounds on generalization error.

Experimental results validate the effectiveness of FeTa on neural networks.

Abstract

Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. We start by proposing a cheap pruning algorithm for fully connected DNN layers based on difference of convex functions (DC) optimisation, that requires little or no retraining. We then provide a theoretical analysis for the growth in the Generalization Error (GE) of a DNN for the case of bounded perturbations to the hidden layers, of which weight pruning is a special case. Our pruning method is orders of magnitude faster than competing approaches, while our theoretical analysis sheds light to previously observed problems in DNN pruning. Experiments on commnon feedforward neural networks validate our results.