Information-Computation Tradeoffs for Learning Margin Halfspaces with Random Classification Noise

TL;DR

This paper investigates the fundamental limits of efficiently learning margin halfspaces with noise, revealing an inherent gap between the optimal sample complexity and what is achievable with polynomial-time algorithms.

Contribution

It establishes a theoretical tradeoff showing that efficient algorithms require more samples than the information-theoretic minimum for learning noisy margin halfspaces.

Findings

Sample complexity is $ ilde{O}(1/(eta^2 au))$ for the problem.

Efficient algorithms have a lower bound of $ ilde{ ilde{ heta}}(1/(eta^{1/2} au^2))$ on sample complexity.

There is an inherent gap between optimal and computationally feasible learning.

Abstract

We study the problem of PAC learning $\gamma$-margin halfspaces with Random Classification Noise. We establish an information-computation tradeoff suggesting an inherent gap between the sample complexity of the problem and the sample complexity of computationally efficient algorithms. Concretely, the sample complexity of the problem is $\widetilde{\Theta}(1/(\gamma^2 \epsilon))$. We start by giving a simple efficient algorithm with sample complexity $\widetilde{O}(1/(\gamma^2 \epsilon^2))$. Our main result is a lower bound for Statistical Query (SQ) algorithms and low-degree polynomial tests suggesting that the quadratic dependence on $1/\epsilon$ in the sample complexity is inherent for computationally efficient algorithms. Specifically, our results imply a lower bound of $\widetilde{\Omega}(1/(\gamma^{1/2} \epsilon^2))$ on the sample complexity of any efficient SQ learner or low-degree test.