Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexity

TL;DR

This paper introduces approximate dimensional and margin complexity measures, demonstrating their necessity and sufficiency for learning with linear predictors and kernels, thus providing better tools to analyze the limitations of such methods.

Contribution

It proposes and studies approximate complexity notions, showing they are both necessary and sufficient for learning, unlike exact variants.

Findings

Approximate complexity measures are necessary for learning with linear models.

These measures provide insights into the limitations of kernel methods.

Approximate notions are sufficient for effective learning.

Abstract

We present and study approximate notions of dimensional and margin complexity, which correspond to the minimal dimension or norm of an embedding required to approximate, rather then exactly represent, a given hypothesis class. We show that such notions are not only sufficient for learning using linear predictors or a kernel, but unlike the exact variants, are also necessary. Thus they are better suited for discussing limitations of linear or kernel methods.