MetricOpt: Learning to Optimize Black-Box Evaluation Metrics

TL;DR

MetricOpt introduces a black-box approach to directly optimize non-differentiable evaluation metrics by learning a differentiable value function, improving performance across various vision tasks without complex loss engineering.

Contribution

It proposes a novel method to optimize arbitrary metrics by learning a differentiable value function, enabling effective metric supervision during model fine-tuning.

Findings

Achieves state-of-the-art results on classification, retrieval, and detection metrics.

Provides consistent improvements over existing methods.

Generalizes well to new tasks and architectures.

Abstract

We study the problem of directly optimizing arbitrary non-differentiable task evaluation metrics such as misclassification rate and recall. Our method, named MetricOpt, operates in a black-box setting where the computational details of the target metric are unknown. We achieve this by learning a differentiable value function, which maps compact task-specific model parameters to metric observations. The learned value function is easily pluggable into existing optimizers like SGD and Adam, and is effective for rapidly finetuning a pre-trained model. This leads to consistent improvements since the value function provides effective metric supervision during finetuning, and helps to correct the potential bias of loss-only supervision. MetricOpt achieves state-of-the-art performance on a variety of metrics for (image) classification, image retrieval and object detection. Solid benefits are found over competing methods, which often involve complex loss design or adaptation. MetricOpt also generalizes well to new tasks and model architectures.