An information theoretic formulation of the Dictionary Learning and Sparse Coding Problems on Statistical Manifolds

TL;DR

This paper introduces an information theoretic approach to dictionary learning and sparse coding on statistical manifolds, avoiding explicit sparsity norms and extending to positive definite matrices, with promising results in vision tasks.

Contribution

It proposes a novel framework that uses KL-centers for sparse coding on statistical manifolds without explicit sparsity norms, extending to symmetric positive definite matrices.

Findings

Effective in classification tasks across computer vision datasets.

Achieves higher sparsity and accuracy compared to state-of-the-art methods.

Validated through extensive experiments and theoretical analysis.

Abstract

In this work, we propose a novel information theoretic framework for dictionary learning (DL) and sparse coding (SC) on a statistical manifold (the manifold of probability distributions). Unlike the traditional DL and SC framework, our new formulation {\it does not explicitly incorporate any sparsity inducing norm in the cost function but yet yields SCs}. Moreover, we extend this framework to the manifold of symmetric positive definite matrices, $\mathcal{P}_n$. Our algorithm approximates the data points, which are probability distributions, by the weighted Kullback-Leibeler center (KL-center) of the dictionary atoms. The KL-center is the minimizer of the maximum KL-divergence between the unknown center and members of the set whose center is being sought. Further, {\it we proved that this KL-center is a sparse combination of the dictionary atoms}. Since, the data reside on a statistical manifold, the data fidelity term can not be as simple as in the case of the vector-space data. We therefore employ the geodesic distance between the data and a sparse approximation of the data element. This cost function is minimized using an acceleterated gradient descent algorithm. An extensive set of experimental results show the effectiveness of our proposed framework. We present several experiments involving a variety of classification problems in Computer Vision applications. Further, we demonstrate the performance of our algorithm by comparing it to several state-of-the-art methods both in terms of classification accuracy and sparsity.