Mixed Membership sub-Gaussian Models

TL;DR

This paper introduces the mixed membership sub-Gaussian model, extending Gaussian mixtures to allow observations to belong to multiple components, with an efficient spectral estimation algorithm and strong theoretical guarantees.

Contribution

It proposes a novel mixed membership extension of Gaussian mixture models with a computationally efficient estimator and provable small estimation error under mild conditions.

Findings

The spectral algorithm accurately estimates mixed memberships.

Theoretical guarantees show vanishing estimation error with high probability.

Experimental results outperform existing methods ignoring mixed memberships.

Abstract

The Gaussian mixture model is widely used in unsupervised learning, owing to its simplicity and interpretability. However, a fundamental limitation of the classical Gaussian mixture model is that it forces each observation to belong to exactly one component. In many practical applications, such as genetics, social network analysis, and text mining, an observation may naturally belong to multiple components or exhibit partial membership in several latent components. To overcome this limitation, we propose the mixed membership sub-Gaussian model, which extends the classical Gaussian mixture framework by allowing each observation to belong to multiple components. This model inherits the interpretability of the classical Gaussian mixture model while offering greater flexibility for capturing complex overlapping structures. We develop an efficient spectral algorithm to estimate the mixed membership of each individual observation, and under mild separation conditions on the component centres, we prove that the estimation error of the per-individual membership vector can be made arbitrarily small with high probability. To our knowledge, this is the first work to provide a computationally efficient estimator with such a vanishing-error guarantee for a mixed-membership extension of the Gaussian mixture model. Extensive experimental studies demonstrate that our method outperforms existing approaches that ignore mixed memberships.