What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

TL;DR

WIMHF is a method that uses sparse autoencoders to interpret human feedback data, revealing diverse preferences and safety concerns, and enabling better data curation and personalization.

Contribution

It introduces WIMHF, a novel autoencoder-based approach to automatically extract interpretable features from human feedback datasets.

Findings

Identifies key human-interpretable features that explain preference signals.

Re-labeling harmful data with WIMHF improves safety by 37%.

Enables personalized preference prediction with annotator-specific feature weights.

Abstract

Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.