Egocentric Video-Language Pretraining @ Ego4D Challenge 2022

TL;DR

This paper introduces an egocentric video-language pretraining approach using the Ego4D dataset, achieving state-of-the-art results on multiple challenging tasks like query, localization, and classification.

Contribution

It pioneers egocentric VLP by leveraging pretraining datasets, objectives, and development sets, enabling transfer to various downstream tasks.

Findings

Achieved 10.46 R@1&IoU@0.3 on NLQ

Obtained 10.33 mAP on Moment Query

74% accuracy on Object State Change Classification

Abstract

In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR). Especially, we exploit the recently released Ego4D dataset \cite{grauman2021ego4d} to pioneer Egocentric VLP from pretraining dataset, pretraining objective, and development set. Based on the above three designs, we develop a pretrained video-language model that is able to transfer its egocentric video-text representation or video-only representation to several video downstream tasks. Our Egocentric VLP achieves 10.46R@1&IoU @0.3 on NLQ, 10.33 mAP on MQ, 74% Acc on OSCC, 0.67 sec error on PNR. The code is available at https://github.com/showlab/EgoVLP.