ExpMRC: Explainability Evaluation for Machine Reading Comprehension

TL;DR

This paper introduces ExpMRC, a new benchmark for evaluating the explainability of machine reading comprehension systems, emphasizing the importance of providing both answers and explanations, and demonstrating current models' limitations.

Contribution

The paper presents ExpMRC, a comprehensive benchmark with evidence annotations across multiple datasets, and evaluates state-of-the-art models' ability to generate explanations alongside answers.

Findings

Models lag behind human performance in explainability

Unsupervised evidence extraction approaches show promise

ExpMRC is a challenging benchmark for future research

Abstract

Achieving human-level performance on some of Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE$^+$, and C$^3$ with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources will be available through https://github.com/ymcui/expmrc