Ensembling Strategies for Answering Natural Questions

TL;DR

This paper explores ensembling techniques for question answering systems, demonstrating a strategy that significantly improves F1 scores on the Natural Questions dataset, surpassing previous state-of-the-art results.

Contribution

It introduces an effective ensembling strategy for question answering models, filling a gap in published ensembling methods and achieving new performance benchmarks.

Findings

F1 score for short answers improved by 2.3 points on NQ dev set

Ensembling strategy outperforms single models and previous SOTA

Provides insights into ensembling techniques for QA systems

Abstract

Many of the top question answering systems today utilize ensembling to improve their performance on tasks such as the Stanford Question Answering Dataset (SQuAD) and Natural Questions (NQ) challenges. Unfortunately most of these systems do not publish their ensembling strategies used in their leaderboard submissions. In this work, we investigate a number of ensembling techniques and demonstrate a strategy which improves our F1 score for short answers on the dev set for NQ by 2.3 F1 points over our single model (which outperforms the previous SOTA by 1.9 F1 points).