EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds

TL;DR

EgoAgent is a unified transformer-based model that jointly learns perception, prediction, and action in egocentric environments, capturing their interdependencies for improved performance across various tasks.

Contribution

The paper introduces EgoAgent, a novel unified model that integrates perception, prediction, and action learning in a single transformer architecture, inspired by the perception-action loop.

Findings

Outperforms existing methods on egocentric tasks

Demonstrates superior future state and motion prediction accuracy

Shows effective joint learning of perception, prediction, and action

Abstract

Learning an agent model that behaves like humans-capable of jointly perceiving the environment, predicting the future, and taking actions from a first-person perspective-is a fundamental challenge in computer vision. Existing methods typically train separate models for these abilities, which fail to capture their intrinsic relationships and prevent them from learning from each other. Inspired by how humans learn through the perception-action loop, we propose EgoAgent, a unified agent model that simultaneously learns to represent, predict, and act within a single transformer. EgoAgent explicitly models the causal and temporal dependencies among these abilities by formulating the task as an interleaved sequence of states and actions. It further introduces a joint embedding-action-prediction architecture with temporally asymmetric predictor and observer branches, enabling synergistic optimization across all three capabilities. Comprehensive evaluations of EgoAgent on representative tasks such as image classification, egocentric future state prediction, and 3D human motion prediction demonstrate the superiority of our method. The code and trained models will be publicly available at https://github.com/zju3dv/EgoAgent.