A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

TL;DR

This paper introduces a novel statistical framework and a new redundancy measure, the K/R ratio, to evaluate residual model redundancy in compressed CNNs, supported by theoretical analysis and empirical validation.

Contribution

It develops a tensor-based formulation of CNNs, reveals a discrepancy in sample complexity indicating redundancy, and proposes the K/R ratio as a new measure for residual model redundancy.

Findings

The K/R ratio effectively measures residual redundancy in compressed CNNs.

Sample complexity analysis uncovers discrepancies indicating remaining redundancy.

Ablation studies validate the usefulness of the new redundancy measure.

Abstract

While recently many designs have been proposed to improve the model efficiency of convolutional neural networks (CNNs) on a fixed resource budget, theoretical understanding of these designs is still conspicuously lacking. This paper aims to provide a new framework for answering the question: Is there still any remaining model redundancy in a compressed CNN? We begin by developing a general statistical formulation of CNNs and compressed CNNs via the tensor decomposition, such that the weights across layers can be summarized into a single tensor. Then, through a rigorous sample complexity analysis, we reveal an important discrepancy between the derived sample complexity and the naive parameter counting, which serves as a direct indicator of the model redundancy. Motivated by this finding, we introduce a new model redundancy measure for compressed CNNs, called the $K/R$ ratio, which further allows for nonlinear activations. The usefulness of this new measure is supported by ablation studies on popular block designs and datasets.