A Quantitative Evaluation of Natural Language Question Interpretation for Question Answering Systems

TL;DR

This paper introduces a detailed evaluation method for natural language question interpretation in QA systems, providing finer insights into system performance beyond traditional answer-based assessments.

Contribution

It proposes a subdivided evaluation approach and an evaluation tool focused on NL question interpretation, enhancing analysis of QA system components.

Findings

Deeper insights into QA system performance through the proposed evaluation method

The evaluation tool effectively analyzes NL question interpretation step

Experiments on benchmark datasets demonstrate improved diagnostic capabilities

Abstract

Systematic benchmark evaluation plays an important role in the process of improving technologies for Question Answering (QA) systems. While currently there are a number of existing evaluation methods for natural language (NL) QA systems, most of them consider only the final answers, limiting their utility within a black box style evaluation. Herein, we propose a subdivided evaluation approach to enable finer-grained evaluation of QA systems, and present an evaluation tool which targets the NL question (NLQ) interpretation step, an initial step of a QA pipeline. The results of experiments using two public benchmark datasets suggest that we can get a deeper insight about the performance of a QA system using the proposed approach, which should provide a better guidance for improving the systems, than using black box style approaches.