Cost and Reward Infused Metric Elicitation

TL;DR

This paper extends metric elicitation methods to incorporate cost and reward considerations, enabling more practical and preference-aligned performance metric selection in machine learning models.

Contribution

It introduces an extension of the DLPME algorithm to include bounded costs and rewards, broadening the scope of metric elicitation beyond accuracy-based metrics.

Findings

The extended algorithm converges quickly to the true metric with synthetic data.

Incorporating costs and rewards improves the relevance of selected metrics.

The approach demonstrates practical applicability in diverse scenarios.

Abstract

In machine learning, metric elicitation refers to the selection of performance metrics that best reflect an individual's implicit preferences for a given application. Currently, metric elicitation methods only consider metrics that depend on the accuracy values encoded within a given model's confusion matrix. However, focusing solely on confusion matrices does not account for other model feasibility considerations such as varied monetary costs or latencies. In our work, we build upon the multiclass metric elicitation framework of Hiranandani et al., extrapolating their proposed Diagonal Linear Performance Metric Elicitation (DLPME) algorithm to account for additional bounded costs and rewards. Our experimental results with synthetic data demonstrate our approach's ability to quickly converge to the true metric.