Sample Complexity of Learning Mixtures of Sparse Linear Regressions

TL;DR

This paper advances the understanding of learning mixtures of sparse linear regressions by developing robust algorithms for multiple signals, handling noise, and removing restrictive assumptions, with connections to Gaussian mixtures and error-correcting codes.

Contribution

It introduces the first robust reconstruction algorithm for multiple signals in noisy settings and removes previous restrictive assumptions, broadening the applicability of mixture regression learning.

Findings

First robust algorithm for multiple signals in noisy settings

Handles non-perfect sparsity with good approximation

Circumvents previous restrictive assumptions

Abstract

In the problem of learning mixtures of linear regressions, the goal is to learn a collection of signal vectors from a sequence of (possibly noisy) linear measurements, where each measurement is evaluated on an unknown signal drawn uniformly from this collection. This setting is quite expressive and has been studied both in terms of practical applications and for the sake of establishing theoretical guarantees. In this paper, we consider the case where the signal vectors are sparse; this generalizes the popular compressed sensing paradigm. We improve upon the state-of-the-art results as follows: In the noisy case, we resolve an open question of Yin et al. (IEEE Transactions on Information Theory, 2019) by showing how to handle collections of more than two vectors and present the first robust reconstruction algorithm, i.e., if the signals are not perfectly sparse, we still learn a good sparse approximation of the signals. In the noiseless case, as well as in the noisy case, we show how to circumvent the need for a restrictive assumption required in the previous work. Our techniques are quite different from those in the previous work: for the noiseless case, we rely on a property of sparse polynomials and for the noisy case, we provide new connections to learning Gaussian mixtures and use ideas from the theory of error-correcting codes.