Video-ToC: Video Tree-of-Cue Reasoning

TL;DR

Video-ToC introduces a tree-of-cue reasoning framework for video understanding, combining structured visual cues, adaptive reward mechanisms, and new datasets to improve reasoning capabilities in Video LLMs.

Contribution

The paper presents a novel tree-of-cue reasoning approach with structured visual cues, adaptive rewards, and new datasets for enhanced video understanding in LLMs.

Findings

Outperforms baselines on six video understanding benchmarks.

Demonstrates improved reasoning and perception in video analysis.

Achieves superior results on a video hallucination benchmark.

Abstract

Existing Video Large Language Models (Video LLMs) struggle with complex video understanding, exhibiting limited reasoning capabilities and potential hallucinations. In particular, these methods tend to perform reasoning solely relying on the pretrained inherent reasoning rationales whilst lacking perception-aware adaptation to the input video content. To address this, we propose \textbf{Video-ToC}, a novel video reasoning framework that enhances video understanding through tree-of-cue reasoning. Specifically, our approach introduces three key innovations: (1) A tree-guided visual cue localization mechanism, which endows the model with enhanced fine-grained perceptual capabilities through structured reasoning patterns; (2) A reasoning-demand reward mechanism, which dynamically adjusts the reward value for reinforcement learning (RL) based on the estimation of reasoning demands, enabling on-demand incentives for more effective reasoning strategies; and (3) An automated annotation pipeline that constructs the Video-ToC-SFT-1k and Video-ToC-RL-2k datasets for supervised fine-tuning (SFT) and RL training, respectively. Extensive evaluations on six video understanding benchmarks and a video hallucination benchmark demonstrate the superiority of Video-ToC over baselines and recent methods. Code is available at https://github.com/qizhongtan/Video-ToC.