Towards an understanding of CNNs: analysing the recovery of activation pathways via Deep Convolutional Sparse Coding

TL;DR

This paper analyzes how deep convolutional sparse coding models can recover activation pathways in neural networks, extending prior guarantees and introducing stripe-sparsity to better understand ReLU effectiveness.

Contribution

It extends existing theoretical guarantees for activation recovery in D-CSC models by incorporating stripe-sparsity and probabilistic analysis, enhancing understanding of ReLU activations.

Findings

Recovery guarantees for higher activation densities

Introduction of stripe-sparsity as a measure of ReLU efficacy

High-probability recovery of true activations in extended models

Abstract

Deep Convolutional Sparse Coding (D-CSC) is a framework reminiscent of deep convolutional neural networks (DCNNs), but by omitting the learning of the dictionaries one can more transparently analyse the role of the activation function and its ability to recover activation paths through the layers. Papyan, Romano, and Elad conducted an analysis of such an architecture, demonstrated the relationship with DCNNs and proved conditions under which the D-CSC is guaranteed to recover specific activation paths. A technical innovation of their work highlights that one can view the efficacy of the ReLU nonlinear activation function of a DCNN through a new variant of the tensor's sparsity, referred to as stripe-sparsity. Using this they proved that representations with an activation density proportional to the ambient dimension of the data are recoverable. We extend their uniform guarantees to a modified model and prove that with high probability the true activation is typically possible to recover for a greater density of activations per layer. Our extension follows from incorporating the prior work on one step thresholding by Schnass and Vandergheynst.