How Does Conversation Length Impact User's Satisfaction? A Case Study of Length-Controlled Conversations with LLM-Powered Chatbots

TL;DR

This study investigates how varying conversation lengths with GPT-4 chatbots influences user satisfaction, revealing that conversation quality remains relatively stable despite different dialogue durations, with mixed user reactions.

Contribution

The paper introduces a method to control conversation length in LLM chatbots and provides empirical insights into its effects on user satisfaction.

Findings

Conversation quality remains stable across different dialogue lengths.

Participants had mixed reactions to conversation length variations.

Changing text length does not necessarily alter conversation quality.

Abstract

Users can discuss a wide range of topics with large language models (LLMs), but they do not always prefer solving problems or getting information through lengthy conversations. This raises an intriguing HCI question: How does instructing LLMs to engage in longer or shorter conversations affect conversation quality? In this paper, we developed two Slack chatbots using GPT-4 with the ability to vary conversation lengths and conducted a user study. Participants asked the chatbots both highly and less conversable questions, engaging in dialogues with 0, 3, 5, and 7 conversational turns. We found that the conversation quality does not differ drastically across different conditions, while participants had mixed reactions. Our study demonstrates LLMs' ability to change conversation length and the potential benefits for users resulting from such changes, but we caution that changes in text form may not necessarily imply changes in quality or content.