The Search Problem in Mixture Models

TL;DR

This paper introduces a general framework and matrix-based algorithm for efficiently learning a single component of a mixture model using side information, improving accuracy and computational complexity over existing methods.

Contribution

It develops a simple, general model for side information and specializes it to key mixture model scenarios, demonstrating improved accuracy and efficiency.

Findings

Enhanced parameter estimation accuracy with informative side information

Reduced computational complexity compared to tensor-based methods

Successful application to real datasets showing practical benefits

Abstract

We consider the task of learning the parameters of a {\em single} component of a mixture model, for the case when we are given {\em side information} about that component, we call this the "search problem" in mixture models. We would like to solve this with computational and sample complexity lower than solving the overall original problem, where one learns parameters of all components. Our main contributions are the development of a simple but general model for the notion of side information, and a corresponding simple matrix-based algorithm for solving the search problem in this general setting. We then specialize this model and algorithm to four common scenarios: Gaussian mixture models, LDA topic models, subspace clustering, and mixed linear regression. For each one of these we show that if (and only if) the side information is informative, we obtain parameter estimates with greater accuracy, and also improved computation complexity than existing moment based mixture model algorithms (e.g. tensor methods). We also illustrate several natural ways one can obtain such side information, for specific problem instances. Our experiments on real data sets (NY Times, Yelp, BSDS500) further demonstrate the practicality of our algorithms showing significant improvement in runtime and accuracy.