Sparse Coding with Multi-Layer Decoders using Variance Regularization

TL;DR

This paper introduces a novel variance regularization method for sparse coding that prevents code collapse without regularizing the decoder, enabling effective training of multi-layer decoders for improved image representation and downstream task performance.

Contribution

The work proposes a new variance-based regularization technique for sparse coding that allows training multi-layer decoders without decoder regularization, enhancing interpretability and performance.

Findings

Decoders learned with our method have interpretable features.

Our approach yields higher quality, sparser reconstructions.

Sparse representations improve denoising and classification in low-data scenarios.

Abstract

Sparse representations of images are useful in many computer vision applications. Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the $l_1$ norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements. In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways to effectively train sparse coding systems with multi-layer decoders since they can model more complex relationships than linear dictionaries. In our experiments with MNIST and natural image patches, we show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality reconstructions with sparser representations when compared to autoencoders with linear dictionaries. Additionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime.