ChronusOmni: Improving Time Awareness of Omni Large Language Models

TL;DR

ChronusOmni is a novel omni large language model that enhances temporal awareness across audio and visual modalities, improving explicit and implicit temporal grounding in videos through unified modeling and reinforcement learning.

Contribution

The paper introduces ChronusOmni, a model with interleaved timestamp tokens and reinforcement learning, plus a new dataset, advancing cross-modal temporal reasoning in large language models.

Findings

Achieves over 30% improvement on ChronusAV dataset.

Sets new state-of-the-art on multiple temporal grounding benchmarks.

Demonstrates strong cross-modal temporal reasoning capabilities.

Abstract

Time awareness is a fundamental ability of omni large language models, especially for understanding long videos and answering complex questions. Previous approaches mainly target vision-language scenarios and focus on the explicit temporal grounding questions, such as identifying when a visual event occurs or determining what event happens at aspecific time. However, they often make insufficient use of the audio modality, and overlook implicit temporal grounding across modalities--for example, identifying what is visually present when a character speaks, or determining what is said when a visual event occurs--despite such cross-modal temporal relations being prevalent in real-world scenarios. In this paper, we propose ChronusOmni, an omni large language model designed to enhance temporal awareness for both explicit and implicit audiovisual temporal grounding. First, we interleave text-based timestamp tokens with visual and audio representations at each time unit, enabling unified temporal modeling across modalities. Second, to enforce correct temporal ordering and strengthen fine-grained temporal reasoning, we incorporate reinforcement learning with specially designed reward functions. Moreover, we construct ChronusAV, a temporally-accurate, modality-complete, and cross-modal-aligned dataset to support the training and evaluation on audiovisual temporal grounding task. Experimental results demonstrate that ChronusOmni achieves state-of-the-art performance on ChronusAV with more than 30% improvement and top results on most metrics upon other temporal grounding benchmarks. This highlights the strong temporal awareness of our model across modalities, while preserving general video and audio understanding capabilities.