When Do Decompositions Help for Machine Reading?

TL;DR

This paper investigates when question decompositions aid machine reading tasks, finding they help in few-shot scenarios but can hinder performance with larger datasets, suggesting models can learn decompositions implicitly with more data.

Contribution

The study provides a comprehensive analysis of the effectiveness of question decompositions in machine reading, highlighting their benefits in low-data settings and limitations in larger datasets.

Findings

Decompositions improve performance in few-shot learning scenarios.

With datasets of hundreds of examples, decompositions are often unhelpful or harmful.

Models can implicitly learn decompositions when given sufficient data.

Abstract

Answering complex questions often requires multi-step reasoning in order to obtain the final answer. Most research into decompositions of complex questions involves open-domain systems, which have shown success in using these decompositions for improved retrieval. In the machine reading setting, however, work to understand when decompositions are helpful is understudied. We conduct experiments on decompositions in machine reading to unify recent work in this space, using a range of models and datasets. We find that decompositions can be helpful in the few-shot case, giving several points of improvement in exact match scores. However, we also show that when models are given access to datasets with around a few hundred or more examples, decompositions are not helpful (and can actually be detrimental). Thus, our analysis implies that models can learn decompositions implicitly even with limited data.