LegendreTron: Uprising Proper Multiclass Loss Learning

TL;DR

LegendreTron introduces a novel method for jointly learning proper multiclass loss functions and probabilities, extending previous binary-focused approaches to handle multiple classes using convex function gradients, and demonstrates superior performance on large-scale benchmarks.

Contribution

The paper proposes LegendreTron, a new approach that extends proper loss learning to multiclass problems by leveraging convex function gradients, enabling joint learning of losses and probabilities.

Findings

Outperforms baseline on large multiclass datasets

Achieves statistical significance in all tested datasets with >10 classes

Effectively learns proper canonical losses for multiclass classification

Abstract

Loss functions serve as the foundation of supervised learning and are often chosen prior to model development. To avoid potentially ad hoc choices of losses, statistical decision theory describes a desirable property for losses known as \emph{properness}, which asserts that Bayes' rule is optimal. Recent works have sought to \emph{learn losses} and models jointly. Existing methods do this by fitting an inverse canonical link function which monotonically maps $\mathbb{R}$ to $[0,1]$ to estimate probabilities for binary problems. In this paper, we extend monotonicity to maps between $\mathbb{R}^{C-1}$ and the projected probability simplex $\tilde{\Delta}^{C-1}$ by using monotonicity of gradients of convex functions. We present {\sc LegendreTron} as a novel and practical method that jointly learns \emph{proper canonical losses} and probabilities for multiclass problems. Tested on a benchmark of domains with up to 1,000 classes, our experimental results show that our method consistently outperforms the natural multiclass baseline under a $t$-test at 99% significance on all datasets with greater than 10 classes.