TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

TL;DR

TAVIS introduces a comprehensive benchmark for evaluating active vision and anticipatory gaze in imitation learning, enabling systematic comparison of approaches across diverse tasks and conditions.

Contribution

The paper presents TAVIS, a new benchmark with task suites, evaluation protocols, and metrics for active-vision imitation learning, including a novel anticipatory gaze metric grounded in cognitive science.

Findings

Active vision generally improves performance but varies by task.

Multi-task policies degrade under distribution shifts.

Imitation policies exhibit anticipatory gaze comparable to humans.

Abstract

Active vision -- where a policy controls its own gaze during manipulation -- has emerged as a key capability for imitation learning, with multiple independent systems demonstrating its benefits in the past year. Yet there is no shared benchmark to compare approaches or quantify what active vision contributes, on which task types, and under what conditions. We introduce TAVIS, evaluation infrastructure for active-vision imitation learning, with two complementary task suites -- TAVIS-Head (5 tasks, global search via pan/tilt necks) and TAVIS-Hands (3 tasks, local occlusion via wrist cameras) -- on two humanoid torso embodiments (GR1T2, Reachy2), built on IsaacLab. TAVIS provides three evaluation primitives: a paired headcam-vs-fixedcam protocol on identical demonstrations; GALT (Gaze-Action Lead Time), a novel metric grounded in cognitive science and HRI that quantifies anticipatory gaze in learned policies; and procedural ID/OOD splits. Baseline experiments with Diffusion Policy and $\pi_0$ reveal that (i) active-vision generally helps, but benefits are task-conditional rather than uniform; (ii) multi-task policies degrade sharply under controlled distribution shifts on both suites; and (iii) imitation alone yields anticipatory gaze, with median lead times comparable to the human teleoperator reference. Code, evaluation scripts, demonstrations (LeRobot v3.0; ~2200 episodes) and trained baselines are released at https://github.com/spiglerg/tavis and https://huggingface.co/tavis-benchmark.