PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation

TL;DR

PersoDPO is a scalable framework that improves persona-grounded dialogue models by automatically optimizing preferences based on automatic evaluation signals, enhancing contextual coherence and personalization.

Contribution

The paper introduces PersoDPO, a novel automatic preference optimization method that leverages automatic evaluation signals to fine-tune dialogue models without manual annotation.

Findings

Outperforms baseline models on the FoCus dataset

Enhances coherence and personalization in dialogue responses

Automates preference pair construction for scalable training

Abstract

Personalization and contextual coherence are two essential components in building effective persona-grounded dialogue systems. These aspects play a crucial role in enhancing user engagement and ensuring responses are more relevant and consistent with user identity. However, recent studies indicate that open-source large language models (LLMs) continue to struggle to generate responses that are both contextually grounded and aligned with persona cues, despite exhibiting strong general conversational abilities like fluency and naturalness. We present PersoDPO, a scalable preference optimisation framework that uses supervision signals from automatic evaluations of responses generated by both closed-source and open-source LLMs to fine-tune dialogue models. The framework integrates evaluation metrics targeting coherence and personalization, along with a length-format compliance feature to promote instruction adherence. These signals are combined to automatically construct high-quality preference pairs without manual annotation, enabling a scalable and reproducible training pipeline. Experiments on the FoCus dataset show that an open-source language model fine-tuned with the PersoDPO framework consistently outperforms strong open-source baselines and a standard Direct Preference Optimization (DPO) variant across multiple evaluation dimensions.