MAMS: Model-Agnostic Module Selection Framework for Video Captioning

TL;DR

This paper introduces a model-agnostic framework for adaptive frame selection and token subset construction in video captioning, improving caption quality by focusing on important visual information.

Contribution

It proposes the first adaptive, model-agnostic module selection framework and an attention masking scheme to enhance video captioning performance.

Findings

Significant performance improvements on benchmark datasets.

Effective selection of relevant visual tokens.

Enhanced attention on important visual features.

Abstract

Multi-modal transformers are rapidly gaining attention in video captioning tasks. Existing multi-modal video captioning methods typically extract a fixed number of frames, which raises critical challenges. When a limited number of frames are extracted, important frames with essential information for caption generation may be missed. Conversely, extracting an excessive number of frames includes consecutive frames, potentially causing redundancy in visual tokens extracted from consecutive video frames. To extract an appropriate number of frames for each video, this paper proposes the first model-agnostic module selection framework in video captioning that has two main functions: (1) selecting a caption generation module with an appropriate size based on visual tokens extracted from video frames, and (2) constructing subsets of visual tokens for the selected caption generation module. Furthermore, we propose a new adaptive attention masking scheme that enhances attention on important visual tokens. Our experiments on three different benchmark datasets demonstrate that the proposed framework significantly improves the performance of three recent video captioning models.