TemporalVLM: Video LLMs for Temporal Reasoning in Long Videos

TL;DR

TemporalVLM is a novel video large language model designed for temporal reasoning and detailed understanding of long videos, integrating a visual encoder, LSTMs, and a new dataset for evaluation.

Contribution

We introduce TemporalVLM, the first to incorporate LSTMs into video LLMs, along with a new dataset for long video temporal reasoning tasks.

Findings

TemporalVLM outperforms previous methods in multiple video understanding tasks.

The model effectively combines local and global temporal cues.

The IndustryASM dataset supports detailed temporal analysis in industrial videos.

Abstract

We introduce TemporalVLM, a video large language model (video LLM) for temporal reasoning and fine-grained understanding in long videos. Our approach includes a visual encoder for mapping a long-term video into features which are time-aware and contain both local and global cues. It first divides an input video into short-term clips, which are jointly encoded with timestamps and fused across overlapping temporal windows into time-sensitive local features. Next, the local features are passed through a bidirectional long short-term memory (BiLSTM) module for global feature aggregation. Moreover, to facilitate the evaluation of TemporalVLM, we present a large-scale long video dataset of industry assembly processes, namely IndustryASM, consisting of videos recorded on factory floors with actions and timestamps annotated by industrial engineers for time and motion studies and temporal action segmentation evaluation. Finally, extensive experiments show that TemporalVLM outperforms previous methods across temporal reasoning and fine-grained understanding tasks, i.e., dense video captioning, temporal video grounding, video highlight detection, and temporal action segmentation. To our best knowledge, our work is the first to incorporate LSTMs into video LLMs.