Are current long-term video understanding datasets long-term?

TL;DR

This paper evaluates whether current long-term video datasets genuinely require long-term reasoning, revealing that existing datasets can often be solved using short-term shortcuts, thus questioning their effectiveness for true long-term action recognition.

Contribution

The authors propose a method to assess dataset suitability for long-term recognition and demonstrate that popular datasets can be solved with short-term information, urging the use of more challenging datasets.

Findings

Existing datasets can be solved with short-term shortcuts

Many datasets do not require true long-term reasoning

Researchers should adopt datasets that necessitate long-term information

Abstract

Many real-world applications, from sport analysis to surveillance, benefit from automatic long-term action recognition. In the current deep learning paradigm for automatic action recognition, it is imperative that models are trained and tested on datasets and tasks that evaluate if such models actually learn and reason over long-term information. In this work, we propose a method to evaluate how suitable a video dataset is to evaluate models for long-term action recognition. To this end, we define a long-term action as excluding all the videos that can be correctly recognized using solely short-term information. We test this definition on existing long-term classification tasks on three popular real-world datasets, namely Breakfast, CrossTask and LVU, to determine if these datasets are truly evaluating long-term recognition. Our study reveals that these datasets can be effectively solved using shortcuts based on short-term information. Following this finding, we encourage long-term action recognition researchers to make use of datasets that need long-term information to be solved.