Red Teaming Language Models for Processing Contradictory Dialogues

TL;DR

This paper introduces a new task and dataset for detecting and correcting contradictions in dialogues, aiming to improve language models' consistency and understanding in conversational AI.

Contribution

It presents a novel contradictory dialogue processing task, a dataset with explanations, and a Red Teaming framework to detect, explain, and modify contradictions in conversations.

Findings

Enhanced detection of contradictory dialogues

Framework provides valid explanations for contradictions

Improved ability to modify and correct dialogues

Abstract

Most language models currently available are prone to self-contradiction during dialogues. To mitigate this issue, this study explores a novel contradictory dialogue processing task that aims to detect and modify contradictory statements in a conversation. This task is inspired by research on context faithfulness and dialogue comprehension, which have demonstrated that the detection and understanding of contradictions often necessitate detailed explanations. We develop a dataset comprising contradictory dialogues, in which one side of the conversation contradicts itself. Each dialogue is accompanied by an explanatory label that highlights the location and details of the contradiction. With this dataset, we present a Red Teaming framework for contradictory dialogue processing. The framework detects and attempts to explain the dialogue, then modifies the existing contradictory content using the explanation. Our experiments demonstrate that the framework improves the ability to detect contradictory dialogues and provides valid explanations. Additionally, it showcases distinct capabilities for modifying such dialogues. Our study highlights the importance of the logical inconsistency problem in conversational AI.