NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

TL;DR

This paper reviews the NeurIPS 2020 EfficientQA competition, highlighting approaches to open-domain question answering that balance accuracy with strict memory constraints, and analyzing system performances and evaluation methods.

Contribution

It provides a comprehensive overview of the competition's organization, reviews top submissions, and discusses the trade-offs in system design under memory limitations.

Findings

Top systems effectively balance retrieval and model parameters.

Memory constraints significantly influence system architecture choices.

Analysis informs future evaluation standards for open-domain QA.

Abstract

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.