EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision

TL;DR

EAGLE introduces a memory-augmented framework for egocentric visual query localization that combines appearance and geometry information, enabling accurate 2D-3D localization and surpassing current benchmarks.

Contribution

The paper proposes a novel episodic memory system integrating appearance and geometry for unified 2D-3D localization in egocentric vision, inspired by avian memory consolidation.

Findings

Achieves state-of-the-art results on Ego4D-VQ benchmark.

Effectively models long- and short-term appearance variations.

Enhances spatial discrimination and contour delineation.

Abstract

Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-ofthe-art performance on the Ego4D-VQ benchmark.