Towards Understanding Generalization of Macro-AUC in Multi-label Learning

TL;DR

This paper investigates the theoretical generalization properties of Macro-AUC in multi-label learning, highlighting the impact of label imbalance and proposing new bounds and algorithms.

Contribution

It characterizes the generalization bounds of algorithms based on surrogate losses for Macro-AUC, emphasizing the role of label imbalance and introducing a new concentration inequality.

Findings

Univariate loss algorithms are more sensitive to label imbalance.

Pairwise and reweighted loss algorithms perform better under imbalance.

Empirical results support the theoretical analysis.

Abstract

Macro-AUC is the arithmetic mean of the class-wise AUCs in multi-label learning and is commonly used in practice. However, its theoretical understanding is far lacking. Toward solving it, we characterize the generalization properties of various learning algorithms based on the corresponding surrogate losses w.r.t. Macro-AUC. We theoretically identify a critical factor of the dataset affecting the generalization bounds: \emph{the label-wise class imbalance}. Our results on the imbalance-aware error bounds show that the widely-used univariate loss-based algorithm is more sensitive to the label-wise class imbalance than the proposed pairwise and reweighted loss-based ones, which probably implies its worse performance. Moreover, empirical results on various datasets corroborate our theory findings. To establish it, technically, we propose a new (and more general) McDiarmid-type concentration inequality, which may be of independent interest.