Learning Intersections of Two Margin Halfspaces under Factorizable Distributions

TL;DR

This paper introduces a new polynomial-time algorithm for learning intersections of two margin halfspaces under factorizable distributions, surpassing previous quasi-polynomial algorithms and breaking the correlational statistical query hardness barrier.

Contribution

The authors develop a novel algorithm that efficiently learns intersections of two margin halfspaces under a broad class of distributions, extending tractability beyond prior distribution-specific results.

Findings

The new algorithm runs in polynomial time in dimension and margin inverse.

It demonstrates a separation between CSQ and SQ complexity classes for this problem.

The approach combines tensor analysis, PCA, and non-convex optimization techniques.

Abstract

Learning intersections of halfspaces is a central problem in Computational Learning Theory. Even for just two halfspaces, it remains a major open question whether learning is possible in polynomial time with respect to the margin $\gamma$ of the data points and their dimensionality $d$. The best-known algorithms run in quasi-polynomial time $d^{O(\log(1/\gamma))}$, and it has been shown that this complexity is unavoidable for any algorithm relying solely on correlational statistical queries (CSQ). In this work, we introduce a novel algorithm that provably circumvents the CSQ hardness barrier. Our approach applies to a broad class of distributions satisfying a natural, previously studied, factorizability assumption. Factorizable distributions lie between distribution-specific and distribution-free settings, and significantly extend previously known tractable cases. Under these distributions, we show that CSQ-based methods still require quasipolynomial time even for weakly learning, whereas our algorithm achieves $poly(d,1/\gamma)$ time by leveraging more general statistical queries (SQ), establishing a strong separation between CSQ and SQ for this simple realizable PAC learning problem. Our result is grounded in a rigorous analysis utilizing a novel duality framework that characterizes the moment tensor structure induced by the marginal distributions. Building on these structural insights, we propose new, efficient learning algorithms. These algorithms combine a refined variant of Jennrich's Algorithm with PCA over random projections of the moment tensor, along with a gradient-descent-based non-convex optimization framework.