A Unified Framework for Rank-based Loss Minimization

TL;DR

This paper introduces a unified optimization framework for rank-based loss functions in machine learning, demonstrating convergence and efficiency through experiments on synthetic and real data.

Contribution

It proposes a novel proximal alternating direction method of multipliers for optimizing diverse rank-based losses, unifying various loss types under one framework.

Findings

The algorithm converges under mild conditions.

It is effective and efficient on synthetic and real datasets.

The framework accommodates both convex and nonconvex rank-based losses.

Abstract

The empirical loss, commonly referred to as the average loss, is extensively utilized for training machine learning models. However, in order to address the diverse performance requirements of machine learning models, the use of the rank-based loss is prevalent, replacing the empirical loss in many cases. The rank-based loss comprises a weighted sum of sorted individual losses, encompassing both convex losses like the spectral risk, which includes the empirical risk and conditional value-at-risk, and nonconvex losses such as the human-aligned risk and the sum of the ranked range loss. In this paper, we introduce a unified framework for the optimization of the rank-based loss through the utilization of a proximal alternating direction method of multipliers. We demonstrate the convergence and convergence rate of the proposed algorithm under mild conditions. Experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed algorithm.