Unit Impulse Response as an Explainer of Redundancy in a Deep Convolutional Neural Network

TL;DR

This paper introduces a novel approach using unit impulse response to analyze and interpret redundancy across the depth of CNNs, providing deeper insights into their internal dynamics and robustness.

Contribution

It proposes a new method to identify and understand systemic redundancy in CNNs across depth using impulse response analysis, enhancing interpretability.

Findings

Redundancy across CNN depth contributes to robustness.

Impulse response can reveal systemic redundancy.

Methods improve understanding of CNN internal dynamics.

Abstract

Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the increased cost of redundant computations. Several methods have been proposed which leverage metrics that quantify the redundancy in each layer. However, layer-wise evaluation in these methods disregards the long-range redundancy which exists across depth on account of the distributed nature of the features learned by the model. In this paper, we propose (i) a mechanism to empirically demonstrate the robustness in performance of a CNN on account of redundancy across its depth, (ii) a method to identify the systemic redundancy in response of a CNN across depth using the understanding of unit impulse response, we subsequently demonstrate use of these methods to interpret redundancy in few networks as example. These techniques provide better insights into the internal dynamics of a CNN