Robust Model Selection and Nearly-Proper Learning for GMMs

TL;DR

This paper introduces a robust, efficient method for model selection in univariate Gaussian mixture models, capable of handling adversarial corruptions and estimating the number of components with provable guarantees.

Contribution

It presents the first polynomial-time algorithm for robustly estimating the number of components in GMMs with provable guarantees, even under adversarial noise.

Findings

Constructs a GMM with ~O(k) components approximating the distribution within ~O(ε)

Works with polynomially many samples, poly(k/ε) time complexity

Extends techniques to Fourier-sparse signal reconstruction

Abstract

In learning theory, a standard assumption is that the data is generated from a finite mixture model. But what happens when the number of components is not known in advance? The problem of estimating the number of components, also called model selection, is important in its own right but there are essentially no known efficient algorithms with provable guarantees let alone ones that can tolerate adversarial corruptions. In this work, we study the problem of robust model selection for univariate Gaussian mixture models (GMMs). Given $\textsf{poly}(k/\epsilon)$ samples from a distribution that is $\epsilon$-close in TV distance to a GMM with $k$ components, we can construct a GMM with $\widetilde{O}(k)$ components that approximates the distribution to within $\widetilde{O}(\epsilon)$ in $\textsf{poly}(k/\epsilon)$ time. Thus we are able to approximately determine the minimum number of components needed to fit the distribution within a logarithmic factor. Prior to our work, the only known algorithms for learning arbitrary univariate GMMs either output significantly more than $k$ components (e.g. $k/\epsilon^2$ components for kernel density estimates) or run in time exponential in $k$. Moreover, by adapting our techniques we obtain similar results for reconstructing Fourier-sparse signals.