Internet-augmented language models through few-shot prompting for open-domain question answering

TL;DR

This paper demonstrates that large language models augmented with web-based retrieval and few-shot prompting significantly improve open-domain question answering, offering a strong, fine-tuning-free baseline that leverages external knowledge sources.

Contribution

It introduces a method to condition language models on web-retrieved information via few-shot prompts without additional training, enhancing open-domain QA performance.

Findings

Web-augmented LMs outperform closed-book models of similar size.

Using multiple retrieved evidences and reranking improves accuracy.

Increasing inference compute benefits smaller models.

Abstract

In this work, we aim to capitalize on the unique few-shot capabilities of large-scale language models (LSLMs) to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models (LMs), which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-time compute.