Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel-Young Losses

TL;DR

This paper introduces a new convex smooth surrogate loss with a linear regret bound for discrete target losses, leveraging Fenchel-Young losses and convolutional negentropy to improve optimization and estimation.

Contribution

It constructs a novel surrogate loss that maintains linear regret bounds despite smoothness, using Fenchel-Young losses and infimal convolution techniques.

Findings

Constructed a convex smooth surrogate loss with linear regret bounds.

Demonstrated the surrogate's equivalence to infimal convolution of negentropy and Bayes risk.

Enabled consistent estimation of class probabilities.

Abstract

Surrogate regret bounds, also known as excess risk bounds, bridge the gap between the convergence rates of surrogate and target losses. The regret transfer is lossless if the surrogate regret bound is linear. While convex smooth surrogate losses are appealing in particular due to the efficient estimation and optimization, the existence of a trade-off between the loss smoothness and linear regret bound has been believed in the community. Under this scenario, the better optimization and estimation properties of convex smooth surrogate losses may inevitably deteriorate after undergoing the regret transfer onto a target loss. We overcome this dilemma for arbitrary discrete target losses by constructing a convex smooth surrogate loss, which entails a linear surrogate regret bound composed with a tailored prediction link. The construction is based on Fenchel--Young losses generated by the convolutional negentropy, which are equivalent to the infimal convolution of a generalized negentropy and the target Bayes risk. Consequently, the infimal convolution enables us to derive a smooth loss while maintaining the surrogate regret bound linear. We additionally benefit from the infimal convolution to have a consistent estimator of the underlying class probability. Our results are overall a novel demonstration of how convex analysis penetrates into optimization and statistical efficiency in risk minimization.