Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection

TL;DR

This paper introduces WikiDYK, a large-scale, evolving benchmark for real-world knowledge injection into language models, revealing that bidirectional models outperform causal models in memorization and proposing a collaborative framework to enhance knowledge reliability.

Contribution

The paper presents WikiDYK, a novel benchmark for knowledge injection, and demonstrates that bidirectional models have superior memorization abilities, along with a modular ensemble framework to improve knowledge reliability.

Findings

Bidirectional LMs outperform causal LMs in knowledge memorization.

Ensemble framework improves knowledge reliability by up to 29.1%.

WikiDYK is a scalable, evolving benchmark for real-world knowledge testing.

Abstract

Despite significant advances in large language models (LLMs), their knowledge memorization capabilities remain underexplored, due to the lack of standardized and high-quality test ground. In this paper, we introduce a novel, real-world and large-scale knowledge injection benchmark that evolves continuously over time without requiring human intervention. Specifically, we propose WikiDYK, which leverages recently-added and human-written facts from Wikipedia's "Did You Know..." entries. These entries are carefully selected by expert Wikipedia editors based on criteria such as verifiability and clarity. Each entry is converted into multiple question-answer pairs spanning diverse task formats from easy cloze prompts to complex multi-hop questions. WikiDYK contains 12,290 facts and 77,180 questions, which is also seamlessly extensible with future updates from Wikipedia editors. Extensive experiments using continued pre-training reveal a surprising insight: despite their prevalence in modern LLMs, Causal Language Models (CLMs) demonstrate significantly weaker knowledge memorization capabilities compared to Bidirectional Language Models (BiLMs), exhibiting a 23% lower accuracy in terms of reliability. To compensate for the smaller scales of current BiLMs, we introduce a modular collaborative framework utilizing ensembles of BiLMs as external knowledge repositories to integrate with LLMs. Experiment shows that our framework further improves the reliability accuracy by up to 29.1%.