Understanding How People Rate Their Conversations

TL;DR

This study investigates how personality traits, especially agreeableness and extraversion, influence user ratings of conversational agents, revealing that agreeableness significantly correlates with higher ratings and that storytelling may impact user perceptions.

Contribution

The paper introduces a method to elicit personality traits during interactions with conversational agents and analyzes their effect on user ratings, highlighting the importance of agreeableness.

Findings

Agreeableness significantly correlates with higher conversation ratings.

Extraversion may influence ratings, but more data is needed.

Users who opt-in to hear a story tend to rate interactions more positively.

Abstract

User ratings play a significant role in spoken dialogue systems. Typically, such ratings tend to be averaged across all users and then utilized as feedback to improve the system or personalize its behavior. While this method can be useful to understand broad, general issues with the system and its behavior, it does not take into account differences between users that affect their ratings. In this work, we conduct a study to better understand how people rate their interactions with conversational agents. One macro-level characteristic that has been shown to correlate with how people perceive their inter-personal communication is personality. We specifically focus on agreeableness and extraversion as variables that may explain variation in ratings and therefore provide a more meaningful signal for training or personalization. In order to elicit those personality traits during an interaction with a conversational agent, we designed and validated a fictional story, grounded in prior work in psychology. We then implemented the story into an experimental conversational agent that allowed users to opt-in to hearing the story. Our results suggest that for human-conversational agent interactions, extraversion may play a role in user ratings, but more data is needed to determine if the relationship is significant. Agreeableness, on the other hand, plays a statistically significant role in conversation ratings: users who are more agreeable are more likely to provide a higher rating for their interaction. In addition, we found that users who opted to hear the story were, in general, more likely to rate their conversational experience higher than those who did not.