VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering

TL;DR

VideoForest introduces a person-anchored hierarchical reasoning framework for cross-video question answering, leveraging person features and multi-level structures to improve understanding and reasoning across multiple video streams.

Contribution

The paper presents a novel person-anchored hierarchical reasoning framework and a new CrossVideoQA benchmark for effective cross-video understanding without end-to-end training.

Findings

Achieved 71.93% accuracy in person recognition

Reached 83.75% in behavior analysis

Attained 51.67% in summarization and reasoning

Abstract

Cross-video question answering presents significant challenges beyond traditional single-video understanding, particularly in establishing meaningful connections across video streams and managing the complexity of multi-source information retrieval. We introduce VideoForest, a novel framework that addresses these challenges through person-anchored hierarchical reasoning. Our approach leverages person-level features as natural bridge points between videos, enabling effective cross-video understanding without requiring end-to-end training. VideoForest integrates three key innovations: 1) a human-anchored feature extraction mechanism that employs ReID and tracking algorithms to establish robust spatiotemporal relationships across multiple video sources; 2) a multi-granularity spanning tree structure that hierarchically organizes visual content around person-level trajectories; and 3) a multi-agent reasoning framework that efficiently traverses this hierarchical structure to answer complex cross-video queries. To evaluate our approach, we develop CrossVideoQA, a comprehensive benchmark dataset specifically designed for person-centric cross-video analysis. Experimental results demonstrate VideoForest's superior performance in cross-video reasoning tasks, achieving 71.93% accuracy in person recognition, 83.75% in behavior analysis, and 51.67% in summarization and reasoning, significantly outperforming existing methods. Our work establishes a new paradigm for cross-video understanding by unifying multiple video streams through person-level features, enabling sophisticated reasoning across distributed visual information while maintaining computational efficiency.