Meta Answering for Machine Reading

TL;DR

This paper introduces a meta question answering framework where a meta-answerer interacts with a black box environment, demonstrating that a simple meta-answerer can outperform the environment in improving answer quality on natural language datasets.

Contribution

It proposes a novel meta-answering framework that enhances machine reading by joint training of answer scoring and context selection, outperforming baseline models.

Findings

Meta-answering improves recall and precision over baseline models.

Humans outperform machine readers with minimal context.

Meta-answerer outperforms the environment on Natural Questions dataset.

Abstract

We investigate a framework for machine reading, inspired by real world information-seeking problems, where a meta question answering system interacts with a black box environment. The environment encapsulates a competitive machine reader based on BERT, providing candidate answers to questions, and possibly some context. To validate the realism of our formulation, we ask humans to play the role of a meta-answerer. With just a small snippet of text around an answer, humans can outperform the machine reader, improving recall. Similarly, a simple machine meta-answerer outperforms the environment, improving both precision and recall on the Natural Questions dataset. The system relies on joint training of answer scoring and the selection of conditioning information.