CDConv: A Benchmark for Contradiction Detection in Chinese Conversations

TL;DR

This paper introduces CDConv, a comprehensive benchmark with 12,000 annotated Chinese conversations to evaluate and improve contradiction detection in dialogue systems, highlighting the importance of context modeling.

Contribution

The work presents a new benchmark dataset for Chinese dialogue contradiction detection, including methods for automatic conversation generation and analysis of chatbot contradictions.

Findings

State-of-the-art chatbots often produce contradictions.

Context modeling is crucial for contradiction detection.

Challenges remain in accurately identifying contradictions.

Abstract

Dialogue contradiction is a critical issue in open-domain dialogue systems. The contextualization nature of conversations makes dialogue contradiction detection rather challenging. In this work, we propose a benchmark for Contradiction Detection in Chinese Conversations, namely CDConv. It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intra-sentence Contradiction, Role Confusion, and History Contradiction. To efficiently construct the CDConv conversations, we devise a series of methods for automatic conversation generation, which simulate common user behaviors that trigger chatbots to make contradictions. We conduct careful manual quality screening of the constructed conversations and show that state-of-the-art Chinese chatbots can be easily goaded into making contradictions. Experiments on CDConv show that properly modeling contextual information is critical for dialogue contradiction detection, but there are still unresolved challenges that require future research.