A Simple Transformer-Based Model for Ego4D Natural Language Queries Challenge

TL;DR

This paper presents a Transformer-based model for natural language queries in ego-centric videos, integrating multiple video features, achieving competitive results in the Ego4D challenge.

Contribution

It introduces a simple yet effective Transformer-based approach leveraging multiple video features for ego-centric video grounding tasks.

Findings

Achieved 12.64% Mean R@1, ranking 2nd in the challenge.

Surpassed top solutions with up to 5.5% higher R@5 at tIoU=0.3.

Demonstrated effectiveness of combining multiple video features with a Transformer model.

Abstract

This report describes Badgers@UW-Madison, our submission to the Ego4D Natural Language Queries (NLQ) Challenge. Our solution inherits the point-based event representation from our prior work on temporal action localization, and develops a Transformer-based model for video grounding. Further, our solution integrates several strong video features including SlowFast, Omnivore and EgoVLP. Without bells and whistles, our submission based on a single model achieves 12.64% Mean R@1 and is ranked 2nd on the public leaderboard. Meanwhile, our method garners 28.45% (18.03%) R@5 at tIoU=0.3 (0.5), surpassing the top-ranked solution by up to 5.5 absolute percentage points.