NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory

TL;DR

NaQ introduces a novel data augmentation method that transforms narrations into training data, significantly improving episodic memory retrieval in egocentric videos and achieving state-of-the-art results on the Ego4D NLQ benchmark.

Contribution

The paper proposes Narrations-as-Queries (NaQ), a new strategy that leverages existing narrations to enhance video query localization models, enabling zero-shot and few-shot learning capabilities.

Findings

NaQ doubles the accuracy of top models on Ego4D NLQ.

NaQ achieves the best results to date on the Ego4D NLQ challenge.

NaQ improves performance on long-tail object category queries.

Abstract

Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand. However, the structured nature of the learning problem (free-form text query inputs, localized video temporal window outputs) and its needle-in-a-haystack nature makes it both technically challenging and expensive to supervise. We introduce Narrations-as-Queries (NaQ), a data augmentation strategy that transforms standard video-text narrations into training data for a video query localization model. Validating our idea on the Ego4D benchmark, we find it has tremendous impact in practice. NaQ improves multiple top models by substantial margins (even doubling their accuracy), and yields the very best results to date on the Ego4D NLQ challenge, soundly outperforming all challenge winners in the CVPR and ECCV 2022 competitions and topping the current public leaderboard. Beyond achieving the state-of-the-art for NLQ, we also demonstrate unique properties of our approach such as the ability to perform zero-shot and few-shot NLQ, and improved performance on queries about long-tail object categories. Code and models: {\small\url{http://vision.cs.utexas.edu/projects/naq}}.