Temporally-Grounded Language Generation: A Benchmark for Real-Time Vision-Language Models

TL;DR

This paper introduces a new benchmark, TGLG, for evaluating real-time vision-language models on their ability to generate temporally aligned language responses to streaming visual input, supported by datasets, a new metric, and a novel model.

Contribution

The paper proposes TGLG, a benchmark for real-time, temporally-grounded language generation, along with datasets, a new evaluation metric TRACE, and a time-synchronized model VLM-TSI.

Findings

VLM-TSI outperforms baseline models in TGLG tasks.

Performance on TGLG remains modest, indicating the challenge of real-time alignment.

The benchmark and model facilitate future research in real-time vision-language systems.

Abstract

Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also precisely timed. We identify two core capabilities necessary for such settings -- $\textit{perceptual updating}$ and $\textit{contingency awareness}$ -- and propose a new benchmark task, $\textbf{Temporally-Grounded Language Generation (TGLG)}$, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, $\textbf{TRACE}$, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present $\textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}$, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest -- highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Code and data available $\href{https://github.com/yukw777/tglg}{here}$.