On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

TL;DR

This paper generalizes basis pursuit to a multi-layer setting, introduces new iterative algorithms for solving it, and connects these algorithms to recurrent CNN architectures that improve performance without increasing parameters.

Contribution

It proposes a novel multi-layer basis pursuit formulation, develops convergent iterative algorithms, and links these to recurrent CNN architectures for improved learning.

Findings

Algorithms converge to near-optimal solutions.

Unfolded algorithms lead to recurrent CNNs.

Recurrent CNNs outperform classical CNNs with same parameters.

Abstract

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.