When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generation

TL;DR

This paper explores how dialogue agents can better incorporate interlocutor profiles to enhance response consistency and speaker recognition, revealing challenges with unfamiliar interlocutors and the effects of information masking.

Contribution

It systematically investigates the role of interlocutor profiles in persona-based dialogue generation and evaluates the impact of information disclosure and fine-tuning.

Findings

Access to interlocutor's persona improves speaker recognition.

Masking interlocutor information reduces recognition accuracy.

Models struggle with unfamiliar interlocutors and topics.

Abstract

Endowing dialogue agents with persona information has proven to significantly improve the consistency and diversity of their generations. While much focus has been placed on aligning dialogues with provided personas, the adaptation to the interlocutor's profile remains largely underexplored. In this work, we investigate three key aspects: (1) a model's ability to align responses with both the provided persona and the interlocutor's; (2) its robustness when dealing with familiar versus unfamiliar interlocutors and topics, and (3) the impact of additional fine-tuning on specific persona-based dialogues. We evaluate dialogues generated with diverse speaker pairings and topics, framing the evaluation as an author identification task and employing both LLM-as-a-judge and human evaluations. By systematically masking or disclosing information about the interlocutor, we assess its impact on dialogue generation. Results show that access to the interlocutor's persona improves the recognition of the target speaker, while masking it does the opposite. Although models generalise well across topics, they struggle with unfamiliar interlocutors. Finally, we found that in zero-shot settings, LLMs often copy biographical details, facilitating identification but trivialising the task.