Exploring the Roles of NLP-based Dialog Indicators in Predicting User Experience in interacting with Large Language Model System

TL;DR

This study explores how NLP-based dialog indicators like coherence and emotion can predict user experience in large language model interactions, offering a new approach to assess and improve chat systems.

Contribution

It introduces a conceptual model linking dialog indicators to user experience factors and validates it using empirical data from 120 participants.

Findings

Dialog indicators can predict chat experience.

Prompts and user intentions influence dialog quality.

User affinity partially mediates the relationship.

Abstract

The use of Large Language Models for dialogue systems is rising, presenting a new challenge: how do we assess users' chat experience in these systems? Leveraging Natural Language Processing (NLP)-powered dialog analyzers to create dialog indicators like Coherence and Emotion has the potential to predict the chat experience. In this paper, we proposed a conceptual model to explain the relationship between the dialog indicators and various factors related to the chat experience, such as users' intentions, affinity toward dialog agents, and prompts of the agents' characters. We evaluated the conceptual model using PLS-SEM with 120 participants and found it well fit. Our results suggest that dialog indicators can predict the chat experience and fully mediate the impact of prompts and user intentions. Additionally, users' affinity toward agents can partially explain these predictions. Our findings demonstrate the potential of using dialog indicators in predicting the chat experience. Through the conceptual model we propose, researchers can apply the dialog analyzers to generate dialog indicators to constantly monitor the dialog process and improve the user's chat experience accordingly.