Performance Metric Elicitation from Pairwise Classifier Comparisons

TL;DR

This paper introduces a method to efficiently determine the performance metric a practitioner values in binary classification by using pairwise classifier comparisons, leveraging geometric properties for robustness and efficiency.

Contribution

It formalizes the metric elicitation problem from pairwise feedback and develops provably query-efficient algorithms for linear and linear-fractional metrics.

Findings

Algorithms are provably query-efficient.

Method is robust to feedback noise.

Applicable to linear and linear-fractional metrics.

Abstract

Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect) classification. In particular, we focus on eliciting binary classification performance metrics from pairwise feedback, where a practitioner is queried to provide relative preference between two classifiers. By exploiting key geometric properties of the space of confusion matrices, we obtain provably query efficient algorithms for eliciting linear and linear-fractional performance metrics. We further show that our method is robust to feedback and finite sample noise.