Learning with Instance Bundles for Reading Comprehension

TL;DR

This paper introduces novel supervision techniques for reading comprehension models that leverage relationships among related question-answer pairs within bundles, improving accuracy significantly.

Contribution

It proposes new contrastive supervision methods using instance bundles, enhancing training signals beyond traditional maximum likelihood estimation.

Findings

Up to 11% accuracy improvement on HotpotQA.

Effective use of related question-answer bundles for training.

Demonstrated benefits on ROPES dataset.

Abstract

When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets -- HotpotQA and ROPES -- showing up to 11% absolute gains in accuracy.