Improving Span-based Question Answering Systems with Coarsely Labeled Data

TL;DR

This paper enhances span-based question answering systems by effectively leveraging coarse-grained paragraph relevance data through latent-variable modeling and novel training objectives, leading to significant performance improvements.

Contribution

It introduces explicit latent-variable models and posterior distillation methods to better utilize coarse annotations in fine-grained QA systems, surpassing standard multi-task learning approaches.

Findings

Latent-variable models outperform standard multi-task learning.

Explicit modeling of fine-coarse relationships yields larger gains.

Coarse supervision significantly improves QA accuracy.

Abstract

We study approaches to improve fine-grained short answer Question Answering models by integrating coarse-grained data annotated for paragraph-level relevance and show that coarsely annotated data can bring significant performance gains. Experiments demonstrate that the standard multi-task learning approach of sharing representations is not the most effective way to leverage coarse-grained annotations. Instead, we can explicitly model the latent fine-grained short answer variables and optimize the marginal log-likelihood directly or use a newly proposed \emph{posterior distillation} learning objective. Since these latent-variable methods have explicit access to the relationship between the fine and coarse tasks, they result in significantly larger improvements from coarse supervision.