Beyond the Low-Degree Algorithm: Mixtures of Subcubes and Their Applications

TL;DR

This paper introduces an efficient algorithm for learning mixtures of subcubes in high-dimensional binary spaces, with applications to decision trees with stochastic transitions, advancing understanding of mixture models and their learnability.

Contribution

It presents a novel $n^{O(\log k)}$-time algorithm based on multilinear moments for learning mixtures of subcubes, and applies this to decision trees with stochastic transitions, improving bounds and understanding.

Findings

Algorithm learns mixtures of subcubes in polynomial time for fixed parameters.

Approximate Bayes optimal classifier within additive error for decision trees with stochastic transitions.

Provides new bounds for learning mixtures of binary product distributions.

Abstract

We introduce the problem of learning mixtures of $k$ subcubes over $\{0,1\}^n$, which contains many classic learning theory problems as a special case (and is itself a special case of others). We give a surprising $n^{O(\log k)}$-time learning algorithm based on higher-order multilinear moments. It is not possible to learn the parameters because the same distribution can be represented by quite different models. Instead, we develop a framework for reasoning about how multilinear moments can pinpoint essential features of the mixture, like the number of components. We also give applications of our algorithm to learning decision trees with stochastic transitions (which also capture interesting scenarios where the transitions are deterministic but there are latent variables). Using our algorithm for learning mixtures of subcubes, we can approximate the Bayes optimal classifier within additive error $\epsilon$ on $k$-leaf decision trees with at most $s$ stochastic transitions on any root-to-leaf path in $n^{O(s + \log k)}\cdot\text{poly}(1/\epsilon)$ time. In this stochastic setting, the classic Occam algorithms for learning decision trees with zero stochastic transitions break down, while the low-degree algorithm of Linial et al. inherently has a quasipolynomial dependence on $1/\epsilon$. In contrast, as we will show, mixtures of $k$ subcubes are uniquely determined by their degree $2 \log k$ moments and hence provide a useful abstraction for simultaneously achieving the polynomial dependence on $1/\epsilon$ of the classic Occam algorithms for decision trees and the flexibility of the low-degree algorithm in being able to accommodate stochastic transitions. Using our multilinear moment techniques, we also give the first improved upper and lower bounds since the work of Feldman et al. for the related but harder problem of learning mixtures of binary product distributions.