Positive Distribution Shift as a Framework for Understanding Tractable Learning

TL;DR

This paper introduces the concept of Positive Distribution Shift (PDS), showing how choosing a suitable training distribution can transform intractable learning problems into tractable ones, emphasizing the computational benefits of PDS.

Contribution

The paper formalizes PDS, demonstrating how it can make certain hard learning problems easier and connecting it with membership query learning, highlighting its computational advantages.

Findings

PDS can turn intractable problems into tractable ones.

Certain hard classes are easily learnable under PDS.

PDS has strong connections with membership query learning.

Abstract

We study a setting where the goal is to learn a target function f(x) with respect to a target distribution D(x), but training is done on i.i.d. samples from a different training distribution D'(x), labeled by the true target f(x). Such a distribution shift (here in the form of covariate shift) is usually viewed negatively, as hurting or making learning harder, and the traditional distribution shift literature is mostly concerned with limiting or avoiding this negative effect. In contrast, we argue that with a well-chosen D'(x), the shift can be positive and make learning easier -- a perspective called Positive Distribution Shift (PDS). Such a perspective is central to contemporary machine learning, where much of the innovation is in finding good training distributions D'(x), rather than changing the training algorithm. We further argue that the benefit is often computational rather than statistical, and that PDS allows computationally hard problems to become tractable even using standard gradient-based training. We formalize different variants of PDS, show how certain hard classes are easily learnable under PDS, and make connections with membership query learning.