An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022

TL;DR

This paper introduces CONE, an efficient coarse-to-fine video alignment framework for natural language queries in videos, leveraging dynamic window slicing, contrastive learning, and pre-trained multi-modal models to improve retrieval accuracy.

Contribution

The paper presents a novel window-centric alignment framework that enhances efficiency and accuracy in video-language retrieval tasks for ego-centric videos.

Findings

Achieved R1@IoU=0.3 of 15.26 on the Ego4D challenge

Achieved R1@IoU=0.5 of 9.24 on the Ego4D challenge

Demonstrated effective coarse-to-fine alignment using contrastive learning and pre-trained models.

Abstract

This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022. We leverage our model CONE, an efficient window-centric COarse-to-fiNE alignment framework. Specifically, CONE dynamically slices the long video into candidate windows via a sliding window approach. Centering at windows, CONE (1) learns the inter-window (coarse-grained) semantic variance through contrastive learning and speeds up inference by pre-filtering the candidate windows relevant to the NL query, and (2) conducts intra-window (fine-grained) candidate moments ranking utilizing the powerful multi-modal alignment ability of the contrastive vision-text pre-trained model EgoVLP. On the blind test set, CONE achieves 15.26 and 9.24 for R1@IoU=0.3 and R1@IoU=0.5, respectively.