Towards Understanding Residual and Dilated Dense Neural Networks via Convolutional Sparse Coding

TL;DR

This paper introduces two novel multi-layer convolutional sparse coding models, Res-CSC and MSD-CSC, providing a theoretical understanding of residual and dilated dense neural networks, and demonstrates their effectiveness through extensive experiments.

Contribution

It proposes new models Res-CSC and MSD-CSC that connect sparse coding with residual and dilated dense networks, offering theoretical insights and practical algorithms.

Findings

Res-CSC models residual networks with sparse coding principles.

MSD-CSC explains dilated and dense connections mathematically.

Both models outperform competing methods on multiple datasets.

Abstract

Convolutional neural network (CNN) and its variants have led to many state-of-art results in various fields. However, a clear theoretical understanding about them is still lacking. Recently, multi-layer convolutional sparse coding (ML-CSC) has been proposed and proved to equal such simply stacked networks (plain networks). Here, we think three factors in each layer of it including the initialization, the dictionary design and the number of iterations greatly affect the performance of ML-CSC. Inspired by these considerations, we propose two novel multi-layer models--residual convolutional sparse coding model (Res-CSC) and mixed-scale dense convolutional sparse coding model (MSD-CSC), which have close relationship with the residual neural network (ResNet) and mixed-scale (dilated) dense neural network (MSDNet), respectively. Mathematically, we derive the shortcut connection in ResNet as a special case of a new forward propagation rule on ML-CSC. We find a theoretical interpretation of the dilated convolution and dense connection in MSDNet by analyzing MSD-CSC, which gives a clear mathematical understanding about them. We implement the iterative soft thresholding algorithm (ISTA) and its fast version to solve Res-CSC and MSD-CSC, which can employ the unfolding operation for further improvements. At last, extensive numerical experiments and comparison with competing methods demonstrate their effectiveness using three typical datasets.