Class Probability Estimation via Differential Geometric Regularization

TL;DR

This paper introduces a geometric regularization method for class probability estimation that penalizes the volume of the probability submanifold to prevent overfitting, applicable to various classifiers with differentiable probability estimators.

Contribution

It proposes a novel geometric regularization technique based on submanifold volume to improve class probability estimation in supervised learning.

Findings

Regularization reduces overfitting by controlling estimator oscillations.

The method improves classification performance over standard regularizers.

Applicable to both binary and multiclass classification tasks.

Abstract

We study the problem of supervised learning for both binary and multiclass classification from a unified geometric perspective. In particular, we propose a geometric regularization technique to find the submanifold corresponding to a robust estimator of the class probability $P(y|\pmb{x})$. The regularization term measures the volume of this submanifold, based on the intuition that overfitting produces rapid local oscillations and hence large volume of the estimator. This technique can be applied to regularize any classification function that satisfies two requirements: firstly, an estimator of the class probability can be obtained; secondly, first and second derivatives of the class probability estimator can be calculated. In experiments, we apply our regularization technique to standard loss functions for classification, our RBF-based implementation compares favorably to widely used regularization methods for both binary and multiclass classification.