Creating, Using and Assessing a Generative-AI-Based Human-Chatbot-Dialogue Dataset with User-Interaction Learning Capabilities

TL;DR

This paper presents a novel dataset of human-AI dialogues generated with ChatGPT 3.5, focusing on specific user language proficiency and emotional states, to enhance dialogue quality and learning capabilities.

Contribution

It introduces a method for creating and evaluating a dialogue dataset with controlled user language and emotion levels, facilitating improved human-AI interaction and learning.

Findings

Generated dialogues meet quality standards

Language complexity is systematically evaluated

Interaction patterns are stored for future analysis

Abstract

The study illustrates a first step towards an ongoing work aimed at developing a dataset of dialogues potentially useful for customer service conversation management between humans and AI chatbots. The approach exploits ChatGPT 3.5 to generate dialogues. One of the requirements is that the dialogue is characterized by a specific language proficiency level of the user; the other one is that the user expresses a specific emotion during the interaction. The generated dialogues were then evaluated for overall quality. The complexity of the language used by both humans and AI agents, has been evaluated by using standard complexity measurements. Furthermore, the attitudes and interaction patterns exhibited by the chatbot at each turn have been stored for further detection of common conversation patterns in specific emotional contexts. The methodology could improve human-AI dialogue effectiveness and serve as a basis for systems that can learn from user interactions.