Neural Conversational QA: Learning to Reason v.s. Exploiting Patterns

TL;DR

This paper investigates how neural conversational QA models learn and exploit dataset patterns, revealing their reliance on spurious clues, and introduces a modified dataset to improve model reasoning capabilities.

Contribution

The paper identifies spurious pattern exploitation in neural QA models and provides a modified dataset to reduce these biases, enhancing reasoning performance.

Findings

Neural models learn spurious dataset patterns.

Heuristic programs exploiting these patterns perform comparably to neural models.

Modified dataset reduces reliance on spurious clues.

Abstract

Neural Conversational QA tasks like ShARC require systems to answer questions based on the contents of a given passage. On studying recent state-of-the-art models on the ShARCQA task, we found indications that the models learn spurious clues/patterns in the dataset. Furthermore, we show that a heuristic-based program designed to exploit these patterns can have performance comparable to that of the neural models. In this paper we share our findings about four types of patterns found in the ShARC corpus and describe how neural models exploit them. Motivated by the aforementioned findings, we create and share a modified dataset that has fewer spurious patterns, consequently allowing models to learn better.