DistinctAD: Distinctive Audio Description Generation in Contexts

TL;DR

DistinctAD is a two-stage framework that improves automatic movie audio description generation by emphasizing distinctiveness through domain adaptation, redundancy reduction, and distinctive word prediction, leading to higher quality narratives.

Contribution

The paper introduces a novel two-stage approach with a CLIP-AD adaptation and innovative modules to enhance the distinctiveness and quality of automatic audio descriptions for movies.

Findings

Outperforms baselines on MAD-Eval, CMD-AD, TV-AD benchmarks

Achieves higher Recall@k/N scores in evaluations

Effectively reduces redundancy and emphasizes distinctive words

Abstract

Audio Descriptions (ADs) aim to provide a narration of a movie in text form, describing non-dialogue-related narratives, such as characters, actions, or scene establishment. Automatic generation of ADs remains challenging due to: i) the domain gap between movie-AD data and existing data used to train vision-language models, and ii) the issue of contextual redundancy arising from highly similar neighboring visual clips in a long movie. In this work, we propose DistinctAD, a novel two-stage framework for generating ADs that emphasize distinctiveness to produce better narratives. To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora, enabling more effective alignment between movie and AD modalities at both global and fine-grained levels. In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context, ensuring the prediction of unique terms specific to the current AD. Comprehensive evaluations on MAD-Eval, CMD-AD, and TV-AD benchmarks demonstrate the superiority of DistinctAD, with the model consistently outperforming baselines, particularly in Recall@k/N, highlighting its effectiveness in producing high-quality, distinctive ADs.