Quadratic Metric Elicitation for Fairness and Beyond

TL;DR

This paper introduces a novel quadratic metric elicitation method that captures complex user preferences for classification fairness and beyond, improving flexibility and robustness over existing linear approaches.

Contribution

It develops a quadratic metric elicitation strategy for multiclass classification, extending to polynomial metrics, with applications in fairness and improved preference modeling.

Findings

Effective in eliciting quadratic group-fairness metrics

Requires only relative preference feedback

Achieves near-optimal query complexity

Abstract

Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for eliciting more flexible multiclass metrics defined by quadratic functions of rates, designed to reflect human preferences better. We show its application in eliciting quadratic violation-based group-fair metrics. Our strategy requires only relative preference feedback, is robust to noise, and achieves near-optimal query complexity. We further extend this strategy to eliciting polynomial metrics -- thus broadening the use cases for metric elicitation.