CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation

TL;DR

CGoDial is a comprehensive large-scale Chinese benchmark for goal-oriented dialog evaluation, covering multiple knowledge sources and realistic spoken scenarios to advance dialog system research.

Contribution

It introduces a new challenging Chinese benchmark with diverse datasets, real conversation data, and varied experimental settings for robust dialog system evaluation.

Findings

Benchmark covers 96,763 sessions and 574,949 turns across three knowledge types.

Includes real conversation data and spoken features to simulate real-world scenarios.

Experimental settings evaluate generalization, adaptability, and robustness of dialog models.

Abstract

Practical dialog systems need to deal with various knowledge sources, noisy user expressions, and the shortage of annotated data. To better solve the above problems, we propose CGoDial, new challenging and comprehensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions and 574,949 dialog turns totally, covering three datasets with different knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with tree-formed knowledge, and a retrieval-based dialog (RBD) dataset with candidate-formed knowledge. To bridge the gap between academic benchmarks and spoken dialog scenarios, we either collect data from real conversations or add spoken features to existing datasets via crowd-sourcing. The proposed experimental settings include the combinations of training with either the entire training set or a few-shot training set, and testing with either the standard test set or a hard test subset, which can assess model capabilities in terms of general prediction, fast adaptability and reliable robustness.