N-best Response-based Analysis of Contradiction-awareness in Neural Response Generation Models

TL;DR

This paper analyzes how well neural response generation models avoid contradictions by examining the consistency of their n-best response lists, especially using polar questions, revealing their strengths and limitations.

Contribution

It introduces a quantitative method to assess contradiction-awareness in neural response models through n-best list analysis using polar questions.

Findings

Recent models show varying levels of contradiction-awareness.

N-best list consistency correlates with contradiction avoidance.

Limitations identified in current contradiction-awareness methods.

Abstract

Avoiding the generation of responses that contradict the preceding context is a significant challenge in dialogue response generation. One feasible method is post-processing, such as filtering out contradicting responses from a resulting n-best response list. In this scenario, the quality of the n-best list considerably affects the occurrence of contradictions because the final response is chosen from this n-best list. This study quantitatively analyzes the contextual contradiction-awareness of neural response generation models using the consistency of the n-best lists. Particularly, we used polar questions as stimulus inputs for concise and quantitative analyses. Our tests illustrate the contradiction-awareness of recent neural response generation models and methodologies, followed by a discussion of their properties and limitations.