Learning transformed product distributions

TL;DR

This paper studies the problem of learning transformed product distributions, showing both the computational hardness for general transformations and providing efficient algorithms for specific cases like sums of Bernoulli variables.

Contribution

It introduces a framework for learning distributions after known transformations, proves hardness results, and presents efficient algorithms for sums of Bernoulli variables.

Findings

Generic algorithms require O(n/^2) samples but may be exponential in runtime.

Learning can be computationally hard even for simple transformations.

Efficient algorithms are provided for sums of Bernoulli variables, with sample complexity independent of n.

Abstract

We consider the problem of learning an unknown product distribution $X$ over $\{0,1\}^n$ using samples $f(X)$ where $f$ is a \emph{known} transformation function. Each choice of a transformation function $f$ specifies a learning problem in this framework. Information-theoretic arguments show that for every transformation function $f$ the corresponding learning problem can be solved to accuracy $\eps$, using $\tilde{O}(n/\eps^2)$ examples, by a generic algorithm whose running time may be exponential in $n.$ We show that this learning problem can be computationally intractable even for constant $\eps$ and rather simple transformation functions. Moreover, the above sample complexity bound is nearly optimal for the general problem, as we give a simple explicit linear transformation function $f(x)=w \cdot x$ with integer weights $w_i \leq n$ and prove that the corresponding learning problem requires $\Omega(n)$ samples. As our main positive result we give a highly efficient algorithm for learning a sum of independent unknown Bernoulli random variables, corresponding to the transformation function $f(x)= \sum_{i=1}^n x_i$. Our algorithm learns to $\eps$-accuracy in poly$(n)$ time, using a surprising poly$(1/\eps)$ number of samples that is independent of $n.$ We also give an efficient algorithm that uses $\log n \cdot \poly(1/\eps)$ samples but has running time that is only $\poly(\log n, 1/\eps).$