Approximating Human Preferences Using a Multi-Judge Learned System

TL;DR

This paper introduces a framework that models diverse human preferences by aggregating outputs from multiple judges conditioned on different rubrics, improving alignment of language models with human values.

Contribution

It presents a novel persona-based aggregation method for preferences and implements two models, GAM and MLP, to enhance reward modeling in LLMs.

Findings

Improved alignment with human preferences.

Robustness against judge biases.

Effective aggregation of diverse judgments.

Abstract

Aligning LLM-based judges with human preferences is a significant challenge, as they are difficult to calibrate and often suffer from rubric sensitivity, bias, and instability. Overcoming this challenge advances key applications, such as creating reliable reward models for Reinforcement Learning from Human Feedback (RLHF) and building effective routing systems that select the best-suited model for a given user query. In this work, we propose a framework for modeling diverse, persona-based preferences by learning to aggregate outputs from multiple rubric-conditioned judges. We investigate the performance of this approach against naive baselines and assess its robustness through case studies on both human and LLM-judges biases. Our primary contributions include a persona-based method for synthesizing preference labels at scale and two distinct implementations of our aggregator: Generalized Additive Model (GAM) and a Multi-Layer Perceptron (MLP).