Classification as Direction Recovery: Improved Guarantees via Scale Invariance

TL;DR

This paper introduces a geometric perspective on binary classification, emphasizing scale invariance and direction recovery, leading to improved theoretical guarantees and a clearer distinction from regression tasks.

Contribution

It establishes a geometric framework showing classification depends on direction rather than scale, improving risk bounds and clarifying the relationship between classification and regression.

Findings

Risk in classification depends only on the direction of the regressor.

The proposed approach improves guarantees for bounding classification risk.

Viewing classification as direction recovery offers new insights into algorithm comparison.

Abstract

Modern algorithms for binary classification rely on an intermediate regression problem for computational tractability. In this paper, we establish a geometric distinction between classification and regression that allows risk in these two settings to be more precisely related. In particular, we note that classification risk depends only on the direction of the regressor, and we take advantage of this scale invariance to improve existing guarantees for how classification risk is bounded by the risk in the intermediate regression problem. Building on these guarantees, our analysis makes it possible to compare algorithms more accurately against each other and suggests viewing classification as unique from regression rather than a byproduct of it. While regression aims to converge toward the conditional expectation function in location, we propose that classification should instead aim to recover its direction.