V-DyKnow: A Dynamic Benchmark for Time-Sensitive Knowledge in Vision Language Models

TL;DR

V-DyKnow introduces a benchmark to evaluate how well vision-language models handle time-sensitive facts, revealing their tendency to produce outdated information and challenges in updating knowledge across modalities.

Contribution

The paper presents V-DyKnow, a new dynamic benchmark for assessing time-sensitive knowledge in vision-language models, including analysis of their knowledge reliability and update capabilities.

Findings

VLMs often output outdated facts reflecting their training data snapshots.

Reliability decreases from textual to visual stimuli, even with correct entity recognition.

Existing methods struggle to effectively update models' knowledge across modalities.

Abstract

Vision-Language Models (VLMs) are trained on data snapshots of documents, including images and texts. Their training data and evaluation benchmarks are typically static, implicitly treating factual knowledge as time-invariant. However, real-world facts are intrinsically time-sensitive and subject to erratic and periodic changes, causing model predictions to become outdated. We present V-DyKnow, a Visual Dynamic Knowledge benchmark for evaluating time-sensitive factual knowledge in VLMs. Using V-DyKnow, we benchmark closed- and open-source VLMs and analyze a) the reliability (correctness and consistency) of model responses across modalities and input perturbations; b) the efficacy of knowledge editing and multi-modal RAG methods for knowledge updates across modalities; and c) the sources of outdated predictions, through data and mechanistic analysis. Our results show that VLMs frequently output outdated facts, reflecting outdated snapshots used in the (pre-)training phase. Factual reliability degrades from textual to visual stimuli, even when entities are correctly recognized. Besides, existing alignment approaches fail to consistently update the models' knowledge across modalities. Together, these findings highlight fundamental limitations in how current VLMs acquire and update time-sensitive knowledge across modalities. We release the benchmark, code, and evaluation data.