Thrust: Adaptively Propels Large Language Models with External Knowledge

TL;DR

This paper introduces Thrust, a novel metric for assessing whether large language models need external knowledge for a specific task, enabling more cost-efficient and effective knowledge retrieval.

Contribution

The paper proposes IAPEK, an adaptive method that uses Thrust to selectively retrieve external knowledge, improving efficiency and performance in large language models.

Findings

Thrust effectively measures model knowledgeability at the instance level.

Using Thrust as a retrieval indicator improves performance by 26% on average.

The approach reduces retrieval costs by 88% of tasks.

Abstract

Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary. To achieve this goal, we propose measuring whether a PTLM contains enough knowledge to solve an instance with a novel metric, Thrust, which leverages the representation distribution of a small number of seen instances. Extensive experiments demonstrate that thrust is a good measurement of PTLM models' instance-level knowledgeability. Moreover, we can achieve significantly higher cost-efficiency with the Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited knowledge-seeking budget due to computation latency or costs.