Classification Performance Metric Elicitation and its Applications
Gaurush Hiranandani

TL;DR
This paper introduces a formal framework for selecting optimal classification metrics based on user preferences, providing robust strategies for eliciting various metrics, including group-fair and complex multiclass metrics, applicable to real-world machine learning tasks.
Contribution
It formalizes metric elicitation as a principled approach, devises novel strategies for eliciting diverse classification metrics, and demonstrates practical applicability including a real-user study.
Findings
Robust elicitation strategies for linear and linear-fractional metrics.
Extension to group-fair and complex multiclass metrics.
Successful real-user validation of the framework.
Abstract
Given a learning problem with real-world tradeoffs, which cost function should the model be trained to optimize? This is the metric selection problem in machine learning. Despite its practical interest, there is limited formal guidance on how to select metrics for machine learning applications. This thesis outlines metric elicitation as a principled framework for selecting the performance metric that best reflects implicit user preferences. Once specified, the evaluation metric can be used to compare and train models. In this manuscript, we formalize the problem of Metric Elicitation and devise novel strategies for eliciting classification performance metrics using pairwise preference feedback over classifiers. Specifically, we provide novel strategies for eliciting linear and linear-fractional metrics for binary and multiclass classification problems, which are then extended to a…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFace and Expression Recognition · Imbalanced Data Classification Techniques · Machine Learning and Algorithms
