A Near-optimal Algorithm for Learning Margin Halfspaces with Massart Noise

TL;DR

This paper introduces a simple, practical, and nearly optimal algorithm for learning margin halfspaces with Massart noise, achieving near-matching sample complexity bounds and improving computational efficiency over prior methods.

Contribution

It presents a computationally efficient algorithm with near-optimal sample complexity for learning margin halfspaces under Massart noise, nearly matching theoretical lower bounds.

Findings

Achieves sample complexity of rac{1}{\u03b3^2 \u03b5^2} for learning halfspaces.

Uses online SGD on convex losses, making the algorithm simple and practical.

Improves upon previous algorithms with higher sample complexity and less efficiency.

Abstract

We study the problem of PAC learning $\gamma$-margin halfspaces in the presence of Massart noise. Without computational considerations, the sample complexity of this learning problem is known to be $\widetilde{\Theta}(1/(\gamma^2 \epsilon))$. Prior computationally efficient algorithms for the problem incur sample complexity $\tilde{O}(1/(\gamma^4 \epsilon^3))$ and achieve 0-1 error of $\eta+\epsilon$, where $\eta<1/2$ is the upper bound on the noise rate. Recent work gave evidence of an information-computation tradeoff, suggesting that a quadratic dependence on $1/\epsilon$ is required for computationally efficient algorithms. Our main result is a computationally efficient learner with sample complexity $\widetilde{\Theta}(1/(\gamma^2 \epsilon^2))$, nearly matching this lower bound. In addition, our algorithm is simple and practical, relying on online SGD on a carefully selected sequence of convex losses.