UniKnow: A Unified Framework for Reliable Language Model Behavior across Parametric and External Knowledge

TL;DR

UniKnow is a comprehensive framework that evaluates and enhances the reliability of language models when integrating parametric and external knowledge across diverse scenarios.

Contribution

The paper introduces UniKnow, a unified framework for assessing and improving language model behavior across various knowledge scenarios, addressing limitations of prior work.

Findings

Existing methods struggle with diverse knowledge scenarios.

Models exhibit scenario-specific biases.

UniKnow enables systematic evaluation and improvement.

Abstract

Language models often benefit from external knowledge beyond parametric knowledge. While this combination enhances performance, achieving reliable knowledge utilization remains challenging, as it requires assessing the state of each knowledge source based on the presence of relevant information. Yet, prior work on knowledge integration often overlooks this challenge by assuming ideal conditions and provides limited coverage of knowledge scenarios. To address this gap, we introduce UniKnow, a Unified framework for reliable LM behavior across parametric and external Knowledge. UniKnow enables controlled evaluation across knowledge scenarios such as knowledge conflict, distraction, and absence conditions that are rarely addressed together. Beyond evaluating existing methods under this setting, we extend our work by introducing UniKnow-Aware methods to support comprehensive evaluation. Experiments on UniKnow reveal that existing methods struggle to generalize across a broader range of knowledge configurations and exhibit scenario-specific biases. UniKnow thus provides a foundation for systematically exploring and improving reliability under knowledge scenarios.