About Time: Advances, Challenges, and Outlooks of Action Understanding

TL;DR

This survey reviews recent progress in video action understanding, highlighting advances, challenges, datasets, and future directions across recognition, prediction, and forecasting tasks in uni- and multi-modal settings.

Contribution

It provides a comprehensive overview of recent developments, challenges, and datasets in action understanding, emphasizing temporal scopes and future research directions.

Findings

Significant performance improvements due to larger datasets and computation.

Diverse applications including scene description, segmentation, synthesis, and context prediction.

Identification of key challenges and future research directions in the field.

Abstract

We have witnessed impressive advances in video action understanding. Increased dataset sizes, variability, and computation availability have enabled leaps in performance and task diversification. Current systems can provide coarse- and fine-grained descriptions of video scenes, extract segments corresponding to queries, synthesize unobserved parts of videos, and predict context across multiple modalities. This survey comprehensively reviews advances in uni- and multi-modal action understanding across a range of tasks. We focus on prevalent challenges, overview widely adopted datasets, and survey seminal works with an emphasis on recent advances. We broadly distinguish between three temporal scopes: (1) recognition tasks of actions observed in full, (2) prediction tasks for ongoing partially observed actions, and (3) forecasting tasks for subsequent unobserved action(s). This division allows us to identify specific action modeling and video representation challenges. Finally, we outline future directions to address current shortcomings.