Simple and Efficient ways to Improve REALM

TL;DR

This paper demonstrates that simple training, supervision, and inference improvements can significantly enhance the performance of the REALM dense retrieval system for open-domain QA, surpassing more complex models.

Contribution

The authors identify undertraining issues in REALM and propose REALM++, a set of simple yet effective improvements that boost accuracy and efficiency without changing model architecture.

Findings

REALM was undertrained during finetuning.

Simple training and inference improvements yield ~5.5% accuracy gains.

REALM++ matches larger models' performance with fewer parameters.

Abstract

Dense retrieval has been shown to be effective for retrieving relevant documents for Open Domain QA, surpassing popular sparse retrieval methods like BM25. REALM (Guu et al., 2020) is an end-to-end dense retrieval system that relies on MLM based pretraining for improved downstream QA efficiency across multiple datasets. We study the finetuning of REALM on various QA tasks and explore the limits of various hyperparameter and supervision choices. We find that REALM was significantly undertrained when finetuning and simple improvements in the training, supervision, and inference setups can significantly benefit QA results and exceed the performance of other models published post it. Our best model, REALM++, incorporates all the best working findings and achieves significant QA accuracy improvements over baselines (~5.5% absolute accuracy) without any model design changes. Additionally, REALM++ matches the performance of large Open Domain QA models which have 3x more parameters demonstrating the efficiency of the setup.