Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion

TL;DR

This paper introduces SeeKeR, a modular language model that integrates internet search, knowledge generation, and response formulation, significantly improving factual accuracy and engagement in dialogue and prompt completion tasks.

Contribution

The paper extends modular language modeling by incorporating internet search as a dedicated module, enhancing factuality and topicality in dialogue and prompt tasks.

Findings

Outperforms BlenderBot 2 in knowledge-grounded conversations

Surpasses GPT2 and GPT3 in factuality and topicality

Demonstrates effectiveness of modular search and generation approach

Abstract

Language models (LMs) have recently been shown to generate more factual responses by employing modularity (Zhou et al., 2021) in combination with retrieval (Adolphs et al., 2021). We extend the recent approach of Adolphs et al. (2021) to include internet search as a module. Our SeeKeR (Search engine->Knowledge->Response) method thus applies a single LM to three modular tasks in succession: search, generating knowledge, and generating a final response. We show that, when using SeeKeR as a dialogue model, it outperforms the state-of-the-art model BlenderBot 2 (Chen et al., 2021) on open-domain knowledge-grounded conversations for the same number of parameters, in terms of consistency, knowledge and per-turn engagingness. SeeKeR applied to topical prompt completions as a standard language model outperforms GPT2 (Radford et al., 2019) and GPT3 (Brown et al., 2020) in terms of factuality and topicality, despite GPT3 being a vastly larger model. Our code and models are made publicly available.