RealTime QA: What's the Answer Right Now?

TL;DR

REALTIME QA is a dynamic, real-time question answering benchmark that evaluates systems on current events, emphasizing the importance of up-to-date information retrieval and updating capabilities.

Contribution

This paper introduces a novel real-time QA benchmark and evaluates large pretrained models, highlighting challenges in handling outdated information and the need for improved retrieval strategies.

Findings

GPT-3 often updates answers with new info from retrieval

Retrieval quality significantly impacts answer accuracy

Identifies the need for systems to detect unanswerable or outdated info

Abstract

We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.