Metric-Optimized Example Weights

TL;DR

This paper introduces a method to learn example weights that optimize complex test metrics directly, improving model performance on diverse datasets and real-world applications by tailoring training to the specific evaluation criteria.

Contribution

It proposes a novel approach to learn example weights that directly optimize arbitrary test metrics, including black box and customized metrics, using a validation set.

Findings

Effective on diverse benchmark datasets

Improves test metric performance in real-world applications

Provides a theoretical generalization bound

Abstract

Real-world machine learning applications often have complex test metrics, and may have training and test data that are not identically distributed. Motivated by known connections between complex test metrics and cost-weighted learning, we propose addressing these issues by using a weighted loss function with a standard loss, where the weights on the training examples are learned to optimize the test metric on a validation set. These metric-optimized example weights can be learned for any test metric, including black box and customized ones for specific applications. We illustrate the performance of the proposed method on diverse public benchmark datasets and real-world applications. We also provide a generalization bound for the method.