What Is Missing: Interpretable Ratings for Large Language Model Outputs

TL;DR

The paper introduces the WIM rating system, which uses natural language feedback to produce interpretable and more effective preference ratings for LLM outputs, improving training signals and enabling qualitative debugging.

Contribution

WIM provides a novel, interpretable rating method using natural language feedback that integrates seamlessly into existing preference learning pipelines.

Findings

WIM yields fewer ties and larger rating differences than numerical ratings.

WIM improves the availability of learning signals in preference data.

WIM enables qualitative debugging through interpretability of feedback.

Abstract

Current Large Language Model (LLM) preference learning methods such as Proximal Policy Optimization and Direct Preference Optimization learn from direct rankings or numerical ratings of model outputs, these rankings are subjective, and a single numerical rating chosen directly by a judge is a poor proxy for the quality of natural language, we introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback, WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method without changing the learning algorithm, to compute a WIM rating, a human or LLM judge writes feedback describing what the model output is missing, we embed the output and the feedback with a sentence embedding model and compute the cosine similarity between the resulting vectors, we empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data, we use interpretable in the following limited sense: for each scalar rating, we can inspect the judge's missing-information text that produced it, enabling qualitative debugging of the preference labels.