Evaluating Contextually Mediated Factual Recall in Multilingual Large Language Models

TL;DR

This paper investigates how multilingual large language models retrieve factual knowledge when the information is embedded in naturalistic contexts rather than explicit queries, revealing that contextual cues often impair recall performance.

Contribution

It introduces a novel evaluation framework for contextually mediated factual recall across languages and analyzes how model size and name types affect retrieval accuracy.

Findings

Contextual mediation degrades factual recall performance.

Larger models are more robust to contextual effects.

Performance varies significantly across different relations.

Abstract

Large language models (LLMs) can recall a wide range of factual knowledge across languages. However, existing factual recall evaluations primarily assess fact retrieval in isolation, where the queried entity is explicitly named and the fact is requested directly. In natural language use, facts are often accessed through context, where the relevant entity is introduced only indirectly. In this work, we study contextually mediated factual recall, asking whether LLMs can reliably retrieve factual knowledge when the target entity is embedded in a naturalistic context rather than queried explicitly, across languages. We construct controlled prompts that preserve the underlying fact while introducing referential mediation through contextual sentences. To disentangle contextual effects from name-specific associations, we further compare performance using synthetic names and real names across languages. Evaluating multiple model families in five languages, we find that contextual mediation consistently degrades factual recall, with substantial variation across relations. Larger models are more robust to contextual mediation, exhibiting a reduced performance gap relative to direct queries, while the effect of real names and name origin is mixed and unsystematic. These findings highlight a gap between isolated factual recall and context-dependent language understanding in multilingual LLMs.