Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks

TL;DR

Temporal2Seq introduces a unified sequence-based framework for multiple temporal video understanding tasks, enabling a single model to perform TAD, TAS, and GEBD with competitive results and strong generalization capabilities.

Contribution

It is the first to unify multiple temporal video understanding tasks into a single sequence-to-sequence framework with a comprehensive co-training dataset.

Findings

Temporal2Seq achieves competitive results across TAD, TAS, and GEBD tasks.

The unified model outperforms single-task models in multi-task settings.

The model demonstrates strong generalization to new datasets from different tasks.

Abstract

With the development of video understanding, there is a proliferation of tasks for clip-level temporal video analysis, including temporal action detection (TAD), temporal action segmentation (TAS), and generic event boundary detection (GEBD). While task-specific video understanding models have exhibited outstanding performance in each task, there remains a dearth of a unified framework capable of simultaneously addressing multiple tasks, which is a promising direction for the next generation of AI. To this end, in this paper, we propose a single unified framework, coined as Temporal2Seq, to formulate the output of these temporal video understanding tasks as a sequence of discrete tokens. With this unified token representation, Temporal2Seq can train a generalist model within a single architecture on different video understanding tasks. In the absence of multi-task learning (MTL) benchmarks, we compile a comprehensive co-training dataset by borrowing the datasets from TAD, TAS, and GEBD tasks. We evaluate our Temporal2Seq generalist model on the corresponding test sets of three tasks, demonstrating that Temporal2Seq can produce reasonable results on various tasks and achieve advantages compared with single-task training on this framework. We also investigate the generalization performance of our generalist model on new datasets from different tasks, which yields superior performance to the specific model.