Moment-Matching Polynomials

TL;DR

This paper introduces a new moment-matching polynomial framework for Boolean functions, enabling low-degree polynomial approximations under broad distributions, and provides the first polynomial-time algorithms for agnostic learning of certain halfspace functions.

Contribution

It presents a novel moment-matching approach that extends polynomial approximation techniques beyond product distributions, leading to new efficient algorithms for learning halfspaces under general distributions.

Findings

First polynomial-time algorithm for agnostically learning functions of constant halfspaces under log-concave distributions

Framework applies to distributions with sub-exponential tails in smoothed analysis setting

Supports the effectiveness of kernel methods like SVMs in practical machine learning scenarios

Abstract

We give a new framework for proving the existence of low-degree, polynomial approximators for Boolean functions with respect to broad classes of non-product distributions. Our proofs use techniques related to the classical moment problem and deviate significantly from known Fourier-based methods, which require the underlying distribution to have some product structure. Our main application is the first polynomial-time algorithm for agnostically learning any function of a constant number of halfspaces with respect to any log-concave distribution (for any constant accuracy parameter). This result was not known even for the case of learning the intersection of two halfspaces without noise. Additionally, we show that in the "smoothed-analysis" setting, the above results hold with respect to distributions that have sub-exponential tails, a property satisfied by many natural and well-studied distributions in machine learning. Given that our algorithms can be implemented using Support Vector Machines (SVMs) with a polynomial kernel, these results give a rigorous theoretical explanation as to why many kernel methods work so well in practice.