Multi-Task Learning of Generalizable Representations for Video Action Recognition

TL;DR

This paper introduces Rev2Net, a multi-task learning framework for video action recognition that enhances generalization across datasets by using auxiliary tasks and a novel training objective, Decoding Discrepancy Penalty.

Contribution

It proposes a new multi-task learning paradigm with auxiliary supervision and a discrepancy penalty to improve cross-dataset generalization in video recognition.

Findings

Rev2Net outperforms baseline models in cross-dataset tests.

Using optical flows as input can harm generalization.

The Decoding Discrepancy Penalty improves feature consistency.

Abstract

In classic video action recognition, labels may not contain enough information about the diverse video appearance and dynamics, thus, existing models that are trained under the standard supervised learning paradigm may extract less generalizable features. We evaluate these models under a cross-dataset experiment setting, as the above label bias problem in video analysis is even more prominent across different data sources. We find that using the optical flows as model inputs harms the generalization ability of most video recognition models. Based on these findings, we present a multi-task learning paradigm for video classification. Our key idea is to avoid label bias and improve the generalization ability by taking data as its own supervision or supervising constraints on the data. First, we take the optical flows and the RGB frames by taking them as auxiliary supervisions, and thus naming our model as Reversed Two-Stream Networks (Rev2Net). Further, we collaborate the auxiliary flow prediction task and the frame reconstruction task by introducing a new training objective to Rev2Net, named Decoding Discrepancy Penalty (DDP), which constraints the discrepancy of the multi-task features in a self-supervised manner. Rev2Net is shown to be effective on the classic action recognition task. It specifically shows a strong generalization ability in the cross-dataset experiments.