PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs

TL;DR

PREF is a novel reference-free framework for evaluating personalized text generation in large language models, combining general quality assessment with user-specific preferences without needing gold references.

Contribution

It introduces a three-step pipeline that separates coverage, preference, and scoring, improving robustness, transparency, and enabling smaller models to approximate larger ones for personalized evaluation.

Findings

PREF outperforms strong baselines in accuracy and calibration.

It achieves closer alignment with human judgments.

The framework enhances robustness and reusability of personalized evaluation.

Abstract

Personalised text generation is essential for user-centric information systems, yet most evaluation methods overlook the individuality of users. We introduce \textbf{PREF}, a \textbf{P}ersonalised \textbf{R}eference-free \textbf{E}valuation \textbf{F}ramework that jointly measures general output quality and user-specific alignment without requiring gold personalised references. PREF operates in a three-step pipeline: (1) a coverage stage uses a large language model (LLM) to generate a comprehensive, query-specific guideline covering universal criteria such as factuality, coherence, and completeness; (2) a preference stage re-ranks and selectively augments these factors using the target user's profile, stated or inferred preferences, and context, producing a personalised evaluation rubric; and (3) a scoring stage applies an LLM judge to rate candidate answers against this rubric, ensuring baseline adequacy while capturing subjective priorities. This separation of coverage from preference improves robustness, transparency, and reusability, and allows smaller models to approximate the personalised quality of larger ones. Experiments on the PrefEval benchmark, including implicit preference-following tasks, show that PREF achieves higher accuracy, better calibration, and closer alignment with human judgments than strong baselines. By enabling scalable, interpretable, and user-aligned evaluation, PREF lays the groundwork for more reliable assessment and development of personalised language generation systems.