Test-Time Adaptation for Video Highlight Detection Using Meta-Auxiliary Learning and Cross-Modality Hallucinations

TL;DR

This paper introduces Highlight-TTA, a novel test-time adaptation framework for video highlight detection that dynamically adjusts models during testing using a meta-auxiliary learning scheme and cross-modality hallucinations, significantly improving generalization and performance.

Contribution

It proposes a new test-time adaptation method with meta-auxiliary learning and cross-modality hallucinations to enhance video highlight detection on unseen videos.

Findings

Improves highlight detection performance across multiple models and datasets.

Enhances model generalization to diverse and unseen test videos.

Achieves superior results compared to baseline methods.

Abstract

Existing video highlight detection methods, although advanced, struggle to generalize well to all test videos. These methods typically employ a generic highlight detection model for each test video, which is suboptimal as it fails to account for the unique characteristics and variations of individual test videos. Such fixed models do not adapt to the diverse content, styles, or audio and visual qualities present in new, unseen test videos, leading to reduced highlight detection performance. In this paper, we propose Highlight-TTA, a test-time adaptation framework for video highlight detection that addresses this limitation by dynamically adapting the model during testing to better align with the specific characteristics of each test video, thereby improving generalization and highlight detection performance. Highlight-TTA is jointly optimized with an auxiliary task, cross-modality hallucinations, alongside the primary highlight detection task. We utilize a meta-auxiliary training scheme to enable effective adaptation through the auxiliary task while enhancing the primary task. During testing, we adapt the trained model using the auxiliary task on the test video to further enhance its highlight detection performance. Extensive experiments with three state-of-the-art highlight detection models and three benchmark datasets show that the introduction of Highlight-TTA to these models improves their performance, yielding superior results.