Using Multiple Samples to Learn Mixture Models

TL;DR

This paper introduces algorithms that leverage multiple samples with different mixing weights to improve the recovery of mixture model parameters, especially in high-dimensional or high-separation scenarios, and extends to topic modeling.

Contribution

It proposes novel algorithms that utilize differences between multiple samples to recover mixture model parameters under milder conditions than existing methods.

Findings

Algorithms outperform previous methods in high-dimensional settings.

Methods enable generalization to unseen words in topic modeling.

Recovery of mixture components is more accurate with multiple samples.

Abstract

In the mixture models problem it is assumed that there are $K$ distributions $\theta_{1},\ldots,\theta_{K}$ and one gets to observe a sample from a mixture of these distributions with unknown coefficients. The goal is to associate instances with their generating distributions, or to identify the parameters of the hidden distributions. In this work we make the assumption that we have access to several samples drawn from the same $K$ underlying distributions, but with different mixing weights. As with topic modeling, having multiple samples is often a reasonable assumption. Instead of pooling the data into one sample, we prove that it is possible to use the differences between the samples to better recover the underlying structure. We present algorithms that recover the underlying structure under milder assumptions than the current state of art when either the dimensionality or the separation is high. The methods, when applied to topic modeling, allow generalization to words not present in the training data.