Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understanding

TL;DR

This paper introduces an adaptive, unsupervised Kernel Temporal Segmentation method for sampling long videos, improving upon uniform sampling and achieving state-of-the-art results in long-form video understanding tasks.

Contribution

The paper proposes a novel, task-agnostic KTS-based sampling approach for long videos, replacing uniform sampling with semantically meaningful segments.

Findings

Consistent performance improvements over existing methods

State-of-the-art results in long-form video classification

Effective in temporal action localization

Abstract

While most modern video understanding models operate on short-range clips, real-world videos are often several minutes long with semantically consistent segments of variable length. A common approach to process long videos is applying a short-form video model over uniformly sampled clips of fixed temporal length and aggregating the outputs. This approach neglects the underlying nature of long videos since fixed-length clips are often redundant or uninformative. In this paper, we aim to provide a generic and adaptive sampling approach for long-form videos in lieu of the de facto uniform sampling. Viewing videos as semantically consistent segments, we formulate a task-agnostic, unsupervised, and scalable approach based on Kernel Temporal Segmentation (KTS) for sampling and tokenizing long videos. We evaluate our method on long-form video understanding tasks such as video classification and temporal action localization, showing consistent gains over existing approaches and achieving state-of-the-art performance on long-form video modeling.