Consistent Multiclass Algorithms for Complex Metrics and Constraints

TL;DR

This paper introduces a unified framework for designing consistent multiclass classifiers optimized for complex metrics and fairness constraints, demonstrating theoretical guarantees and superior empirical performance.

Contribution

It provides a general optimization-based framework for multiclass learning with complex metrics and constraints, including convergence rates and multiple instantiations.

Findings

Algorithms achieve asymptotic consistency.

Framework applies to metrics like G-mean and F1-measure.

Experiments show improved performance over baselines.

Abstract

We present consistent algorithms for multiclass learning with complex performance metrics and constraints, where the objective and constraints are defined by arbitrary functions of the confusion matrix. This setting includes many common performance metrics such as the multiclass G-mean and micro F1-measure, and constraints such as those on the classifier's precision and recall and more recent measures of fairness discrepancy. We give a general framework for designing consistent algorithms for such complex design goals by viewing the learning problem as an optimization problem over the set of feasible confusion matrices. We provide multiple instantiations of our framework under different assumptions on the performance metrics and constraints, and in each case show rates of convergence to the optimal (feasible) classifier (and thus asymptotic consistency). Experiments on a variety of multiclass classification tasks and fairness-constrained problems show that our algorithms compare favorably to the state-of-the-art baselines.