Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences

TL;DR

This paper introduces a robust method for learning evaluator preferences that minimizes assumptions about preference models, ensuring accurate preference learning in diverse real-world evaluation scenarios.

Contribution

The paper proposes a minimally assumption-based algorithm for preference learning that is robust to model mismatch and theoretically guarantees performance.

Findings

The algorithm accurately learns preferences in synthetic and real-world data.

Model mismatch can significantly impair preference learning if not properly addressed.

The method performs well even when traditional linearity assumptions are violated.

Abstract

In many applications, human and LLM evaluators use assessments of relevant criteria to create an overall evaluation for an item or individual. For example, in admissions, committees assess candidates on attributes such as test scores, GPA, and research experience to evaluate their overall fit for the program. Another example arises in medical care where clinicians use patient reports of symptoms to consider preliminary diagnoses and assess risks. Each setting involves mapping multiple criteria to an overall evaluation -- a process that reflects the evaluator's underlying preferences. We focus on the fundamental question of learning these preferences. Many applications of this problem make specific modeling assumptions on evaluator preferences that may be substantially violated in the real world. We make the minimal assumption that the preference function is coordinate-wise non-decreasing, which is reasonable in a large number of evaluation settings. We theoretically characterize the severity of model mismatch for many common assumptions and show that it can lead to significant issues for learning evaluator preferences and other important downstream tasks. We then present an algorithm for learning evaluators' preferences that is robust to model mismatch. We prove theoretically that our algorithm can learn any preference function without sacrificing performance when the linearity assumption holds. Evaluations of our algorithm with synthetic simulations and real-world data confirm its ability to learn preferences robustly and illustrate key aspects of LLM and human preferences.