Sparse coding and dictionary learning based on the MDL principle

TL;DR

This paper introduces a parameter-free framework for sparse coding and dictionary learning based on the MDL principle, addressing statistical properties like overfitting and underfitting, and demonstrating effectiveness in image denoising and classification.

Contribution

It develops a novel MDL-based approach for sparse coding and dictionary learning that is fully parameter-free and can incorporate prior information.

Findings

Effective in image denoising tasks

Improves classification accuracy

Addresses overfitting and underfitting issues

Abstract

The power of sparse signal coding with learned dictionaries has been demonstrated in a variety of applications and fields, from signal processing to statistical inference and machine learning. However, the statistical properties of these models, such as underfitting or overfitting given sets of data, are still not well characterized in the literature. This work aims at filling this gap by means of the Minimum Description Length (MDL) principle -- a well established information-theoretic approach to statistical inference. The resulting framework derives a family of efficient sparse coding and modeling (dictionary learning) algorithms, which by virtue of the MDL principle, are completely parameter free. Furthermore, such framework allows to incorporate additional prior information in the model, such as Markovian dependencies, in a natural way. We demonstrate the performance of the proposed framework with results for image denoising and classification tasks.