The Optimality of Kernel Classifiers in Sobolev Space

TL;DR

This paper analyzes the statistical performance of kernel classifiers in Sobolev spaces, establishing their optimality and extending insights to neural networks, with practical methods for estimating smoothness from data.

Contribution

It provides the first minimax lower bounds for kernel classifiers in Sobolev spaces and demonstrates their optimality, also extending results to neural network classifiers.

Findings

Derived an upper bound on classification excess risk.

Established minimax lower bounds showing optimality.

Proposed a practical method to estimate smoothness from data.

Abstract

Kernel methods are widely used in machine learning, especially for classification problems. However, the theoretical analysis of kernel classification is still limited. This paper investigates the statistical performances of kernel classifiers. With some mild assumptions on the conditional probability $\eta(x)=\mathbb{P}(Y=1\mid X=x)$, we derive an upper bound on the classification excess risk of a kernel classifier using recent advances in the theory of kernel regression. We also obtain a minimax lower bound for Sobolev spaces, which shows the optimality of the proposed classifier. Our theoretical results can be extended to the generalization error of overparameterized neural network classifiers. To make our theoretical results more applicable in realistic settings, we also propose a simple method to estimate the interpolation smoothness of $2\eta(x)-1$ and apply the method to real datasets.