Identifying Auxiliary or Adversarial Tasks Using Necessary Condition Analysis for Adversarial Multi-task Video Understanding

TL;DR

This paper introduces a novel multi-task learning framework that uses Necessary Condition Analysis to differentiate auxiliary and adversarial tasks, improving action recognition in videos by discouraging scene recognition.

Contribution

It proposes Adversarial Multi-Task Neural Networks (AMT), a new framework that employs NCA to identify and penalize adversarial tasks, enhancing video action recognition performance.

Findings

Improves action recognition accuracy by approximately 3%.

Encourages models to focus on action features over scene correlations.

Introduces challenging scene-invariant test splits for evaluation.

Abstract

There has been an increasing interest in multi-task learning for video understanding in recent years. In this work, we propose a generalized notion of multi-task learning by incorporating both auxiliary tasks that the model should perform well on and adversarial tasks that the model should not perform well on. We employ Necessary Condition Analysis (NCA) as a data-driven approach for deciding what category these tasks should fall in. Our novel proposed framework, Adversarial Multi-Task Neural Networks (AMT), penalizes adversarial tasks, determined by NCA to be scene recognition in the Holistic Video Understanding (HVU) dataset, to improve action recognition. This upends the common assumption that the model should always be encouraged to do well on all tasks in multi-task learning. Simultaneously, AMT still retains all the benefits of multi-task learning as a generalization of existing methods and uses object recognition as an auxiliary task to aid action recognition. We introduce two challenging Scene-Invariant test splits of HVU, where the model is evaluated on action-scene co-occurrences not encountered in training. We show that our approach improves accuracy by ~3% and encourages the model to attend to action features instead of correlation-biasing scene features.