Learning Kernel-Based Halfspaces with the Zero-One Loss

TL;DR

This paper introduces a new algorithm for agnostically learning kernel-based halfspaces directly with zero-one loss, providing finite sample guarantees and analyzing its computational complexity.

Contribution

It presents the first finite-time algorithm for learning kernel-based halfspaces with zero-one loss and establishes computational hardness results under cryptographic assumptions.

Findings

Algorithm learns in polynomial time for fixed parameters.

Guarantees the learned classifier is within epsilon of the optimal.

Proves hardness results indicating limits of efficient learning.

Abstract

We describe and analyze a new algorithm for agnostically learning kernel-based halfspaces with respect to the \emph{zero-one} loss function. Unlike most previous formulations which rely on surrogate convex loss functions (e.g. hinge-loss in SVM and log-loss in logistic regression), we provide finite time/sample guarantees with respect to the more natural zero-one loss function. The proposed algorithm can learn kernel-based halfspaces in worst-case time $\poly(\exp(L\log(L/\epsilon)))$, for $\emph{any}$ distribution, where $L$ is a Lipschitz constant (which can be thought of as the reciprocal of the margin), and the learned classifier is worse than the optimal halfspace by at most $\epsilon$. We also prove a hardness result, showing that under a certain cryptographic assumption, no algorithm can learn kernel-based halfspaces in time polynomial in $L$.