Automated Concept Discovery for LLM-as-a-Judge Preference Analysis

TL;DR

This paper introduces an automated method for discovering interpretable concepts behind LLM preference judgments, revealing biases and trends without relying on predefined bias categories, thus enhancing understanding of LLM evaluation behavior.

Contribution

The study presents a novel autoencoder-based approach for extracting interpretable preference features from LLM judgments, outperforming other methods in interpretability and maintaining competitive prediction accuracy.

Findings

LLMs prefer refusing sensitive requests more than humans.

Biases toward concreteness and empathy in LLM judgments.

Biases against legal guidance involving active steps.

Abstract

Large Language Models (LLMs) are increasingly used as scalable evaluators of model outputs, but their preference judgments exhibit systematic biases and can diverge from human evaluations. Prior work on LLM-as-a-judge has largely focused on a small, predefined set of hypothesized biases, leaving open the problem of automatically discovering unknown drivers of LLM preferences. We address this gap by studying several embedding-level concept extraction methods for analyzing LLM judge behavior. We compare these methods in terms of interpretability and predictiveness, finding that sparse autoencoder-based approaches recover substantially more interpretable preference features than alternatives while remaining competitive in predicting LLM decisions. Using over 27k paired responses from multiple human preference datasets and judgments from three LLMs, we analyze LLM judgments and compare them to those of human annotators. Our method both validates existing results, such as the tendency for LLMs to prefer refusal of sensitive requests at higher rates than humans, and uncovers new trends across both general and domain-specific datasets, including biases toward responses that emphasize concreteness and empathy in approaching new situations, toward detail and formality in academic advice, and against legal guidance that promotes active steps like calling police and filing lawsuits. Our results show that automated concept discovery enables systematic analysis of LLM judge preferences without predefined bias taxonomies.