Estimating more camera poses for ego-centric videos is essential for VQ3D

TL;DR

This paper introduces an improved camera pose estimation pipeline for VQ3D in egocentric videos, significantly boosting success rates by optimizing existing frameworks for better accuracy and efficiency.

Contribution

The authors develop a new camera pose estimation pipeline and optimize the VQ3D framework, achieving a twofold increase in success rate over previous baselines.

Findings

Top-1 success rate of 25.8% on VQ3D leaderboard

Enhanced camera pose estimation improves query accuracy

Optimized framework doubles success rate compared to baseline

Abstract

Visual queries 3D localization (VQ3D) is a task in the Ego4D Episodic Memory Benchmark. Given an egocentric video, the goal is to answer queries of the form "Where did I last see object X?", where the query object X is specified as a static image, and the answer should be a 3D displacement vector pointing to object X. However, current techniques use naive ways to estimate the camera poses of video frames, resulting in a low query with pose (QwP) ratio, thus a poor overall success rate. We design a new pipeline for the challenging egocentric video camera pose estimation problem in our work. Moreover, we revisit the current VQ3D framework and optimize it in terms of performance and efficiency. As a result, we get the top-1 overall success rate of 25.8% on VQ3D leaderboard, which is two times better than the 8.7% reported by the baseline.