Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

TL;DR

This paper explores prompt-based methods for real-time game video commentary generation, emphasizing timing and content relevance without fine-tuning, and introduces a dynamic interval decoding strategy that improves alignment with human speech timing.

Contribution

It introduces a novel dynamic interval-based decoding approach for pause-aware commentary generation using multimodal LLMs without fine-tuning.

Findings

Dynamic interval decoding improves timing accuracy.

Prompting alone can support real-time, well-timed commentary.

The approach works effectively across multiple languages and game genres.

Abstract

Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.