GuideLLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing

TL;DR

This paper introduces GuideLLM, a framework for LLM-guided conversations focusing on goal navigation, context management, and empathetic engagement, evaluated through interviews and autobiography generation.

Contribution

It proposes a novel LLM-guided conversation installation and evaluates it with comprehensive automatic and human assessments, outperforming existing models.

Findings

GuideLLM outperforms baseline LLMs in automatic evaluation.

GuideLLM achieves higher human ratings in conversation quality.

The framework effectively manages goals, context, and empathy in dialogues.

Abstract

Although Large Language Models (LLMs) succeed in human-guided conversations such as instruction following and question answering, the potential of LLM-guided conversations-where LLMs direct the discourse and steer the conversation's objectives-remains under-explored. In this study, we first characterize LLM-guided conversation into three fundamental components: (i) Goal Navigation; (ii) Context Management; (iii) Empathetic Engagement, and propose GuideLLM as an installation. We then implement an interviewing environment for the evaluation of LLM-guided conversation. Specifically, various topics are involved in this environment for comprehensive interviewing evaluation, resulting in around 1.4k turns of utterances, 184k tokens, and over 200 events mentioned during the interviewing for each chatbot evaluation. We compare GuideLLM with 6 state-of-the-art LLMs such as GPT-4o and Llama-3-70b-Instruct, from the perspective of interviewing quality, and autobiography generation quality. For automatic evaluation, we derive user proxies from multiple autobiographies and employ LLM-as-a-judge to score LLM behaviors. We further conduct a human-involved experiment by employing 45 human participants to chat with GuideLLM and baselines. We then collect human feedback, preferences, and ratings regarding the qualities of conversation and autobiography. Experimental results indicate that GuideLLM significantly outperforms baseline LLMs in automatic evaluation and achieves consistent leading performances in human ratings.