How Many Answers Should I Give? An Empirical Study of Multi-Answer Reading Comprehension

TL;DR

This paper systematically analyzes the multi-answer reading comprehension phenomenon, categorizing instances, evaluating current models, and proposing strategies to leverage different paradigms for improved performance.

Contribution

It introduces a taxonomy for multi-answer MRC instances, analyzes dataset challenges, and explores combining paradigms with generation models for better answers.

Findings

Different paradigms excel at capturing question key information or context relationships.

Generation models can effectively integrate multiple paradigms.

Annotated datasets and code are released for future research.

Abstract

The multi-answer phenomenon, where a question may have multiple answers scattered in the document, can be well handled by humans but is challenging enough for machine reading comprehension (MRC) systems. Despite recent progress in multi-answer MRC, there lacks a systematic analysis of how this phenomenon arises and how to better address it. In this work, we design a taxonomy to categorize commonly-seen multi-answer MRC instances, with which we inspect three multi-answer datasets and analyze where the multi-answer challenge comes from. We further analyze how well different paradigms of current multi-answer MRC models deal with different types of multi-answer instances. We find that some paradigms capture well the key information in the questions while others better model the relationship between questions and contexts. We thus explore strategies to make the best of the strengths of different paradigms. Experiments show that generation models can be a promising platform to incorporate different paradigms. Our annotations and code are released for further research.