Multitask Learning to Improve Egocentric Action Recognition

TL;DR

This paper demonstrates that multitask learning with auxiliary tasks like verb/noun recognition, hand localization, and gaze prediction improves egocentric action recognition accuracy, outperforming state-of-the-art methods.

Contribution

It introduces a multitask learning framework that leverages secondary tasks to enhance egocentric action recognition without extra test-time inputs.

Findings

Consistent performance improvements on EPIC-Kitchens and EGTEA Gaze+ datasets.

Outperforms state-of-the-art in egocentric action recognition by 3.84%.

Accurately estimates hand and gaze positions as side tasks.

Abstract

In this work we employ multitask learning to capitalize on the structure that exists in related supervised tasks to train complex neural networks. It allows training a network for multiple objectives in parallel, in order to improve performance on at least one of them by capitalizing on a shared representation that is developed to accommodate more information than it otherwise would for a single task. We employ this idea to tackle action recognition in egocentric videos by introducing additional supervised tasks. We consider learning the verbs and nouns from which action labels consist of and predict coordinates that capture the hand locations and the gaze-based visual saliency for all the frames of the input video segments. This forces the network to explicitly focus on cues from secondary tasks that it might otherwise have missed resulting in improved inference. Our experiments on EPIC-Kitchens and EGTEA Gaze+ show consistent improvements when training with multiple tasks over the single-task baseline. Furthermore, in EGTEA Gaze+ we outperform the state-of-the-art in action recognition by 3.84%. Apart from actions, our method produces accurate hand and gaze estimations as side tasks, without requiring any additional input at test time other than the RGB video clips.