Egocentric Video Task Translation

TL;DR

EgoTask Translation (EgoT2) is a unified framework that translates outputs of models trained on different egocentric video tasks, improving performance across multiple tasks by capturing synergies and mitigating competition.

Contribution

EgoT2 introduces a novel task translation approach with separate backbones and a shared translator, enabling multi-task learning and transfer in egocentric video understanding.

Findings

Achieves top-ranked results on four Ego4D benchmark challenges.

Outperforms existing transfer paradigms in egocentric video tasks.

Demonstrates effectiveness across diverse video understanding tasks.

Abstract

Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e.g., classifying sports in one dataset, tracking animals in another). However, in wearable cameras, the immersive egocentric perspective of a person engaging with the world around them presents an interconnected web of video understanding tasks -- hand-object manipulations, navigation in the space, or human-human interactions -- that unfold continuously, driven by the person's goals. We argue that this calls for a much more unified approach. We propose EgoTask Translation (EgoT2), which takes a collection of models optimized on separate tasks and learns to translate their outputs for improved performance on any or all of them at once. Unlike traditional transfer or multi-task learning, EgoT2's flipped design entails separate task-specific backbones and a task translator shared across all tasks, which captures synergies between even heterogeneous tasks and mitigates task competition. Demonstrating our model on a wide array of video tasks from Ego4D, we show its advantages over existing transfer paradigms and achieve top-ranked results on four of the Ego4D 2022 benchmark challenges.