Uncovering Factor Level Preferences to Improve Human-Model Alignment

TL;DR

This paper introduces PROFILE, a framework for uncovering and measuring factor-level preference alignment in large language models, revealing discrepancies and guiding improvements in alignment with human preferences.

Contribution

The paper presents PROFILE, a novel automated method for analyzing factor-level preferences in LLMs, enabling targeted improvements in human-model alignment.

Findings

LLMs show poor factor-level alignment in text generation tasks.

Strong alignment is observed in discrimination tasks.

Leveraging the generation-discrimination gap improves LLM alignment.

Abstract

Large language models (LLMs) often exhibit tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. While crucial for improvement, identifying the factors driving these misalignments remains challenging due to existing evaluation methods' reliance on coarse-grained comparisons and lack of explainability. To address this, we introduce PROFILE, an automated framework to uncover and measure factor-level preference alignment of humans and LLMs. Using PROFILE, we analyze preference alignment across three key tasks: summarization, instruction-following, and document-based QA. We find a significant discrepancy: while LLMs show poor factor-level alignment with human preferences when generating texts, they demonstrate strong alignment in discrimination tasks. We demonstrate how leveraging the identified generation-discrimination gap can be used to improve LLM alignment through multiple approaches, including fine-tuning with self-guidance. Our work highlights the value of factor-level analysis for identifying hidden misalignments and provides a practical framework for improving LLM-human preference alignment.