Online AUC Optimization Based on Second-order Surrogate Loss

TL;DR

This paper introduces a novel second-order surrogate loss for online AUC optimization that improves theoretical regret bounds and demonstrates superior efficiency and effectiveness in large-scale, imbalanced classification tasks.

Contribution

The paper proposes a new second-order surrogate loss and an efficient online algorithm for AUC optimization, achieving tighter regret bounds and extending to nonlinear kernel methods.

Findings

Achieves a regret bound of O(ln T), tighter than existing methods.

Demonstrates superior performance on benchmark datasets.

Extends framework to nonlinear kernel-based models.

Abstract

The Area Under the Curve (AUC) is an important performance metric for classification tasks, particularly in class-imbalanced scenarios. However, minimizing the AUC presents significant challenges due to the non-convex and discontinuous nature of pairwise 0/1 losses, which are difficult to optimize, as well as the substantial memory cost of instance-wise storage, which creates bottlenecks in large-scale applications. To overcome these challenges, we propose a novel second-order surrogate loss based on the pairwise hinge loss, and develop an efficient online algorithm. Unlike conventional approaches that approximate each individual pairwise 0/1 loss term with an instance-wise surrogate function, our approach introduces a new paradigm that directly substitutes the entire aggregated pairwise loss with a surrogate loss function constructed from the first- and second-order statistics of the training data. Theoretically, while existing online AUC optimization algorithms typically achieve an $\mathcal{O}(\sqrt{T})$ regret bound, our method attains a tighter $\mathcal{O}(\ln T)$ bound. Furthermore, we extend the proposed framework to nonlinear settings through a kernel-based formulation. Extensive experiments on multiple benchmark datasets demonstrate the superior efficiency and effectiveness of the proposed second-order surrogate loss in optimizing online AUC performance.