PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Users

TL;DR

This study evaluates personalized language models through large-scale human experiments, revealing that preference fine-tuning outperforms prompting but may have long-term drawbacks, and highlights differences between human and simulated user interactions.

Contribution

It provides empirical evidence comparing context-based prompting and weight-based fine-tuning using real users, and analyzes the divergence between human and simulated user behaviors.

Findings

Preference fine-tuning significantly outperforms generic models and prompting.

Fine-tuning amplifies biases and behaviors like sycophancy and relationship-seeking.

Simulated users do not accurately replicate individual human judgments and behaviors.

Abstract

Personalisation is a standard feature of conversational AI systems used by millions; yet, the efficacy of personalisation methods is often evaluated in academic research using simulated users rather than real people. This raises questions about how users and their simulated counterparts differ in interaction patterns and judgements, as well as whether personalisation is best achieved through context-based prompting or weight-based fine-tuning. Here, in a large-scale within-subject experiment, we re-recruit 530 participants from 52 countries two years after they gave their preferences in the PRISM dataset (Kirk et al., 2024) to evaluate personalised and non-personalised language models in blinded multi-turn conversations. We find preference fine-tuning (P-DPO, Li et al., 2024) significantly outperforms both a generic model and personalised prompting but adapting to individual preference data yields marginal gains over training on pooled preferences from a diverse population. Beyond length biases, fine-tuning amplifies sycophancy and relationship-seeking behaviours that people reward in short-term evaluations but which may introduce deleterious long-term consequences. Replicating this within-subject experiment with simulated users recovers aggregate model hierarchies but simulators perform far below human self-consistency baselines for individual judgements, discuss different topics, exhibit amplified position biases, and produce feedback dynamics that diverge from humans.