On Agnostic PAC Learning using $\mathcal{L}_2$-polynomial Regression and Fourier-based Algorithms

TL;DR

This paper introduces a Hilbert space framework for analyzing agnostic PAC learning, demonstrating that certain regression-based algorithms can achieve near-optimal generalization bounds for specific hypothesis classes.

Contribution

It develops a novel Hilbert space approach to analyze PAC learning, connecting it with least-squares methods like $\\mathcal{L}_2$ polynomial regression and low-degree algorithms.

Findings

Achieves generalization error up to 2 times the optimal error under certain conditions.

Provides tight bounds on generalization error when the optimal error is small.

Revisits PAC learning using regression and Fourier-based methods within the Hilbert space framework.

Abstract

We develop a framework using Hilbert spaces as a proxy to analyze PAC learning problems with structural properties. We consider a joint Hilbert space incorporating the relation between the true label and the predictor under a joint distribution $D$. We demonstrate that agnostic PAC learning with 0-1 loss is equivalent to an optimization in the Hilbert space domain. With our model, we revisit the PAC learning problem using methods based on least-squares such as $\mathcal{L}_2$ polynomial regression and Linial's low-degree algorithm. We study learning with respect to several hypothesis classes such as half-spaces and polynomial-approximated classes (i.e., functions approximated by a fixed-degree polynomial). We prove that (under some distributional assumptions) such methods obtain generalization error up to $2opt$ with $opt$ being the optimal error of the class. Hence, we show the tightest bound on generalization error when $opt\leq 0.2$.