TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models

TL;DR

This paper introduces TPRU, a large-scale dataset and training methodology that significantly improves temporal and procedural understanding in small multimodal models, enabling better real-world embodied AI applications.

Contribution

The paper presents TPRU, a novel dataset with tasks designed for temporal reasoning, and a reinforcement learning fine-tuning approach that enhances small multimodal models' understanding capabilities.

Findings

TPRU-7B accuracy increased from 50.33% to 75.70%.

State-of-the-art performance on TPRU-Test.

Significant improvements on existing benchmarks.

Abstract

Multimodal Large Language Models (MLLMs), particularly smaller, deployable variants, exhibit a critical deficiency in understanding temporal and procedural visual data, a bottleneck hindering their application in real-world embodied AI. This gap is largely caused by a systemic failure in training paradigms, which lack large-scale, procedurally coherent data. To address this problem, we introduce TPRU, a large-scale dataset sourced from diverse embodied scenarios such as robotic manipulation and GUI navigation. TPRU is systematically designed to cultivate temporal reasoning through three complementary tasks: Temporal Reordering, Next-Frame Prediction, and Previous-Frame Review. A key feature is the inclusion of challenging negative samples, compelling models to transition from passive observation to active, cross-modal validation. We leverage TPRU with a reinforcement learning (RL) fine-tuning methodology, specifically targeting the enhancement of resource-efficient models. Experiments show our approach yields dramatic gains: on our manually curated TPRU-Test, the accuracy of TPRU-7B soars from 50.33\% to 75.70\%, a state-of-the-art result that significantly outperforms vastly larger baselines, including GPT-4o. Crucially, these capabilities generalize effectively, demonstrating substantial improvements on established benchmarks. The codebase is available at https://github.com/Stephen-gzk/TPRU/ .