A complete characterization of optimal dictionaries for least squares representation

TL;DR

This paper provides a complete theoretical characterization and practical algorithms for constructing optimal dictionaries that minimize the average Euclidean norm of coefficients in representations, focusing on $ ext{l}_2$-optimality.

Contribution

It introduces a complete characterization of $ ext{l}_2$-optimal dictionaries using rank-1 decompositions and majorization theory, along with polynomial-time construction algorithms.

Findings

Complete characterization of $ ext{l}_2$-optimal dictionaries.

Polynomial-time algorithms for dictionary construction.

Theoretical link between rank-1 decompositions and optimality.

Abstract

Dictionaries are collections of vectors used for representations of elements in Euclidean spaces. While recent research on optimal dictionaries is focussed on providing sparse (i.e., $\ell_0$-optimal,) representations, here we consider the problem of finding optimal dictionaries such that representations of samples of a random vector are optimal in an $\ell_2$-sense. For us, optimality of representation is equivalent to minimization of the average $\ell_2$-norm of the coefficients used to represent the random vector, with the lengths of the dictionary vectors being specified a priori. With the help of recent results on rank-$1$ decompositions of symmetric positive semidefinite matrices and the theory of majorization, we provide a complete characterization of $\ell_2$-optimal dictionaries. Our results are accompanied by polynomial time algorithms that construct $\ell_2$-optimal dictionaries from given data.