A Survey of Machine Narrative Reading Comprehension Assessments

TL;DR

This paper reviews existing machine narrative reading comprehension assessments, proposing a typology based on narrative and comprehension theories to guide future task design and address current challenges.

Contribution

It introduces a comprehensive typology of assessment tasks grounded in narrative and comprehension theories, aiding the development of better benchmarks.

Findings

Proposes a typology capturing key differences among assessment tasks.

Highlights implications for designing new narrative comprehension tasks.

Discusses challenges in evaluating machine narrative understanding.

Abstract

As the body of research on machine narrative comprehension grows, there is a critical need for consideration of performance assessment strategies as well as the depth and scope of different benchmark tasks. Based on narrative theories, reading comprehension theories, as well as existing machine narrative reading comprehension tasks and datasets, we propose a typology that captures the main similarities and differences among assessment tasks; and discuss the implications of our typology for new task design and the challenges of narrative reading comprehension.