Design and Challenges of Cloze-Style Reading Comprehension Tasks on Multiparty Dialogue

TL;DR

This paper examines the limitations of current cloze-style reading comprehension evaluations on multiparty dialogue and introduces new tasks and models to better assess understanding of personal entities in conversations.

Contribution

It identifies evaluation issues in existing methods, proposes more challenging tasks with multiple variables, and develops deep learning models to address these tasks.

Findings

Replacing random splits with chronological splits reduces accuracy from 72% to 34%.

New tasks with multiple variables extend previous single-variable completion.

Deep learning models validate the proposed tasks and highlight ongoing challenges.

Abstract

This paper analyzes challenges in cloze-style reading comprehension on multiparty dialogue and suggests two new tasks for more comprehensive predictions of personal entities in daily conversations. We first demonstrate that there are substantial limitations to the evaluation methods of previous work, namely that randomized assignment of samples to training and test data substantially decreases the complexity of cloze-style reading comprehension. According to our analysis, replacing the random data split with a chronological data split reduces test accuracy on previous single-variable passage completion task from 72\% to 34\%, that leaves much more room to improve. Our proposed tasks extend the previous single-variable passage completion task by replacing more character mentions with variables. Several deep learning models are developed to validate these three tasks. A thorough error analysis is provided to understand the challenges and guide the future direction of this research.