PersoBench: Benchmarking Personalized Response Generation in Large Language Models

TL;DR

This paper introduces PersoBench, an automated benchmarking pipeline to evaluate the personalization ability of large language models in persona-aware dialogue generation, revealing current models' strengths and weaknesses.

Contribution

The paper presents a novel automated benchmarking framework, PersoBench, for assessing LLMs' personalization in dialogue, addressing a gap in existing evaluation methods.

Findings

LLMs generate fluent and diverse responses effectively.

Current LLMs struggle with personalization and coherence.

Evaluation across multiple models and datasets highlights these limitations.

Abstract

While large language models (LLMs) have exhibited impressive conversational capabilities, their proficiency in delivering personalized responses remains unclear. Although recent benchmarks automatically evaluate persona consistency in role-playing contexts using LLM-based judgment, the evaluation of personalization in response generation remains underexplored. To address this gap, we present an automated benchmarking pipeline, PersoBench, to evaluate the personalization ability of LLMs in persona-aware dialogue generation within a zero-shot setting. Our framework employs a structured pipeline comprising speaker-aware annotation, task-specific and context-driven prompt construction, response post-processing, and automated evaluation across multiple dimensions of generation quality. In particular, the pipeline performs text preprocessing and speaker labeling, constructs structured prompts with task instructions and LLM roles, validates response format, and evaluates valid outputs across fluency, personalization, diversity, and coherence. We assess the performance of four open-source and four closed-source LLMs using well-known datasets and a range of explicit metrics. Our findings reveal that while LLMs excel at generating fluent and diverse responses, they are far from satisfactory in delivering personalized and coherent responses, considering both the conversation context and the provided personas.