Relatively Smart: A New Approach for Instance-Optimal Learning

TL;DR

This paper introduces a new framework called relatively smart learning that addresses limitations in semi-supervised learning by focusing on certifiable guarantees, achieving near-optimal sample complexity in distribution-free settings.

Contribution

It proposes the relatively smart learning framework, which relaxes traditional guarantees to bypass impossibility results and analyzes its effectiveness across different distribution settings.

Findings

OIG learner is relatively smart up to a quadratic factor in sample complexity.

Relatively smart learning can be impossible or require specialized approaches in certain distribution families.

The framework successfully bypasses prior impossibility results in distribution-free settings.

Abstract

We revisit the framework of Smart PAC learning, which seeks supervised learners which compete with semi-supervised learners that are provided full knowledge of the marginal distribution on unlabeled data. Prior work has shown that such marginal-by-marginal guarantees are possible for "most" marginals, with respect to an arbitrary fixed and known measure, but not more generally. We discover that this failure can be attributed to an "indistinguishability" phenomenon: There are marginals which cannot be statistically distinguished from other marginals that require different learning approaches. In such settings, semi-supervised learning cannot certify its guarantees from unlabeled data, rendering them arguably non-actionable. We propose relatively smart learning, a new framework which demands that a supervised learner compete only with the best "certifiable" semi-supervised guarantee. We show that such modest relaxation suffices to bypass the impossibility results from prior work. In the distribution-free setting, we show that the OIG learner is relatively smart up to squaring the sample complexity, and show that no supervised learning algorithm can do better. For distribution-family settings, we show that relatively smart learning can be impossible or can require idiosyncratic learning approaches, and its difficulty can be non-monotone in the inclusion order on distribution families.