Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models

TL;DR

This paper explores how large language models integrate their internal parametric knowledge with external contextual information, revealing tendencies to suppress internal knowledge and highlighting challenges in reliably combining these knowledge sources.

Contribution

The study introduces ECHOQA, a new benchmark for evaluating LLMs' handling of different knowledge interactions and analyzes their ability to leverage parametric and contextual knowledge.

Findings

LLMs often suppress parametric knowledge when external context is present.

Tailored instructions can improve reliance on internal knowledge.

LLMs struggle to fully utilize parametric knowledge even with guidance.

Abstract

Large language models (LLMs) encode vast amounts of knowledge during pre-training (parametric knowledge, or PK) and can further be enhanced by incorporating contextual knowledge (CK). Can LLMs effectively integrate their internal PK with external CK to solve complex problems? In this paper, we investigate the dynamic interaction between PK and CK, categorizing their relationships into four types: Supportive, Complementary, Conflicting, and Irrelevant. To support this investigation, we introduce ECHOQA, a benchmark spanning scientific, factual, and commonsense knowledge. Our results show that LLMs tend to suppress their PK when contextual information is available, even when it is complementary or irrelevant. While tailored instructions can encourage LLMs to rely more on their PK, they still struggle to fully leverage it. These findings reveal a key vulnerability in LLMs, raising concerns about their reliability in knowledge-intensive tasks. Resources are available at https://github.com/sitaocheng/Knowledge_Interplay