Predicting with Distributions

TL;DR

This paper introduces a new learning model where the joint distribution over pairs is determined by an unknown function mapping inputs to distributions, and provides reductions to existing PAC learning algorithms for efficient learning.

Contribution

It presents a novel model linking input functions to output distributions and offers general reductions to existing PAC learning algorithms for this setting.

Findings

Reduces the new model to PAC learning of functions and distributions

Provides efficient algorithms via randomized reduction and Le Cam's method

Applicable to a broad class of distribution-based learning problems

Abstract

We consider a new learning model in which a joint distribution over vector pairs $(x,y)$ is determined by an unknown function $c(x)$ that maps input vectors $x$ not to individual outputs, but to entire {\em distributions\/} over output vectors $y$. Our main results take the form of rather general reductions from our model to algorithms for PAC learning the function class and the distribution class separately, and show that virtually every such combination yields an efficient algorithm in our model. Our methods include a randomized reduction to classification noise and an application of Le Cam's method to obtain robust learning algorithms.