Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations

TL;DR

This paper critically examines ChatGPT's ability to generate and annotate high-quality goal-oriented dialogues across different types, modes, and languages, showing human-level quality through extensive evaluations.

Contribution

It provides an in-depth analysis of ChatGPT's performance in generating and annotating goal-oriented dialogues, highlighting its potential as a tool for data creation in complex scenarios.

Findings

ChatGPT-generated dialogues are comparable to human-produced ones in quality.

Performance is consistent across task-oriented, collaborative, and explanatory dialogues.

Multilingual capabilities include English and Italian with similar quality levels.

Abstract

Large pre-trained language models have exhibited unprecedented capabilities in producing high-quality text via prompting techniques. This fact introduces new possibilities for data collection and annotation, particularly in situations where such data is scarce, complex to gather, expensive, or even sensitive. In this paper, we explore the potential of these models to generate and annotate goal-oriented dialogues, and conduct an in-depth analysis to evaluate their quality. Our experiments employ ChatGPT, and encompass three categories of goal-oriented dialogues (task-oriented, collaborative, and explanatory), two generation modes (interactive and one-shot), and two languages (English and Italian). Based on extensive human-based evaluations, we demonstrate that the quality of generated dialogues and annotations is on par with those generated by humans.