Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

TL;DR

Ego-R1 introduces a reinforcement learning-based framework utilizing a structured chain-of-tool-thought process for reasoning over ultra-long egocentric videos, enabling complex multi-modal understanding over extended time periods.

Contribution

The paper presents a novel RL-trained agent with a structured reasoning process and a new dataset for ultra-long egocentric video question answering.

Findings

Effective reasoning over videos spanning a week.

Significant improvement in temporal coverage and understanding.

Demonstrated superiority over baseline methods.

Abstract

We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.