Metritocracy: Representative Metrics for Lite Benchmarks

TL;DR

This paper formalizes and analyzes methods for selecting representative subsets of evaluation metrics for large language models and other applications, using social choice theory to ensure fair and comprehensive representation.

Contribution

It introduces formal notions of positional representation and proportionality for metric subset selection, providing theoretical bounds and practical case studies.

Findings

Bounds on the number of metrics needed for representation

Formalization of metric subset selection using social choice theory

Case studies demonstrating practical applications

Abstract

A common problem in LLM evaluation is how to choose a subset of metrics from a full suite of possible metrics. Subset selection is usually done for efficiency or interpretability reasons, and the goal is often to select a ``representative'' subset of metrics. However, ``representative'' is rarely clearly defined. In this work, we use ideas from social choice theory to formalize two notions of representation for the selection of a subset of evaluation metrics. We first introduce positional representation, which guarantees every alternative is sufficiently represented at every position cutoff. We then introduce positional proportionality, which guarantees no alternative is proportionally over- or under-represented by more than a small error at any position. We prove upper and lower bounds on the smallest number of metrics needed to guarantee either of these properties in the worst case. We also study a generalized form of each property that allows for additional input on groups of metrics that must be represented. Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation.