Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices

TL;DR

This paper systematically benchmarks loss functions and algorithmic choices for deep AUROC optimization, revealing insights into their performance and guiding practitioners in selecting suitable methods for imbalanced classification tasks.

Contribution

It provides the first comprehensive benchmarking of loss functions and algorithmic strategies for deep AUROC maximization, highlighting the effectiveness of composite loss and key algorithmic factors.

Findings

Composite loss functions outperform pairwise loss in convergence and generalization.

Higher positive sampling rates benefit AUROC maximization.

Normalization techniques like sigmoid and $ ext{l}_2$ normalization improve performance.

Abstract

The area under the ROC curve (AUROC) has been vigorously applied for imbalanced classification and moreover combined with deep learning techniques. However, there is no existing work that provides sound information for peers to choose appropriate deep AUROC maximization techniques. In this work, we fill this gap from three aspects. (i) We benchmark a variety of loss functions with different algorithmic choices for deep AUROC optimization problem. We study the loss functions in two categories: pairwise loss and composite loss, which includes a total of 10 loss functions. Interestingly, we find composite loss, as an innovative loss function class, shows more competitive performance than pairwise loss from both training convergence and testing generalization perspectives. Nevertheless, data with more corrupted labels favors a pairwise symmetric loss. (ii) Moreover, we benchmark and highlight the essential algorithmic choices such as positive sampling rate, regularization, normalization/activation, and optimizers. Key findings include: higher positive sampling rate is likely to be beneficial for deep AUROC maximization; different datasets favors different weights of regularizations; appropriate normalization techniques, such as sigmoid and $\ell_2$ score normalization, could improve model performance. (iii) For optimization aspect, we benchmark SGD-type, Momentum-type, and Adam-type optimizers for both pairwise and composite loss. Our findings show that although Adam-type method is more competitive from training perspective, but it does not outperform others from testing perspective.