EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks

TL;DR

EgoTL introduces a think-aloud data collection pipeline for egocentric household tasks, enabling benchmarking and fine-tuning of foundation models to improve long-horizon reasoning and spatial grounding.

Contribution

It presents EgoTL, a novel egocentric data capture method with step-by-step goals and reasoning, and demonstrates its effectiveness in benchmarking and enhancing foundation models for household tasks.

Findings

Foundation models still struggle with egocentric reasoning and spatial grounding.

EgoTL enables benchmarking across six task dimensions and over 100 household tasks.

Fine-tuning with human CoT improves long-horizon planning and spatial reasoning.

Abstract

Large foundation models have made significant advances in embodied intelligence, enabling synthesis and reasoning over egocentric input for household tasks. However, VLM-based auto-labeling is often noisy because the primary data sources lack accurate human action labels, chain-of-thought (CoT), and spatial annotations; these errors are amplified during long-horizon spatial instruction following. These issues stem from insufficient coverage of minute-long, daily household planning tasks and from inaccurate spatial grounding. As a result, VLM reasoning chains and world-model synthesis can hallucinate objects, skip steps, or fail to respect real-world physical attributes. To address these gaps, we introduce EgoTL. EgoTL builds a think-aloud capture pipeline for egocentric data. It uses a say-before-act protocol to record step-by-step goals and spoken reasoning with word-level timestamps, then calibrates physical properties with metric-scale spatial estimators, a memory-bank walkthrough for scene context, and clip-level tags for navigation instructions and detailed manipulation actions. With EgoTL, we are able to benchmark VLMs and World Models on six task dimensions from three layers and long-horizon generation over minute-long sequences across over 100 daily household tasks. We find that foundation models still fall short as egocentric assistants or open-world simulators. Finally, we finetune foundation models with human CoT aligned with metric labels on the training split of EgoTL, which improves long-horizon planning and reasoning, step-wise reasoning, instruction following, and spatial grounding.