Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction

TL;DR

This paper develops a taxonomy of user follow-up query patterns in conversational search, analyzes their motivations and actions, and demonstrates how these patterns correlate with user satisfaction using an LLM-based classifier.

Contribution

It introduces a novel taxonomy of follow-up query patterns, builds an LLM-powered classifier for scalable analysis, and explores correlations with user satisfaction in conversational search.

Findings

Certain follow-up query types signal dissatisfaction, such as Clarifying Queries and Excluding Conditions.

The LLM classifier achieved 73% accuracy in categorizing follow-up queries.

Analysis of real-world logs reveals patterns linked to user satisfaction and dissatisfaction.

Abstract

With large language models (LLMs), conversational search engines shift how users retrieve information from the web by enabling natural conversations to express their search intents over multiple turns. Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results to understand user experience with the system. However, it is underexplored how and why users ask follow-up queries to continue conversations with conversational search engines and how the follow-up queries signal users' satisfaction. From qualitative analysis of 250 conversational turns from an in-lab user evaluation of Naver Cue:, a commercial conversational search engine, we propose a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: (1) users' motivations behind continuing conversations (N = 7) and (2) actions of follow-up queries (N = 11). Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses. To analyze conversational search logs with our taxonomy in a scalable and efficient manner, we built an LLM-powered classifier (73% accuracy). With our classifier, we analyzed 2,061 conversational tuples collected from real-world usage logs of Cue: and examined how the conversation patterns from our taxonomy correlates with satisfaction. Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries. We envision our approach could contribute to automated evaluation of conversation search experience by providing satisfaction signals and grounds for realistic user simulations.