AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description

TL;DR

AutoAD-Zero introduces a training-free method leveraging off-the-shelf visual-language and language models to generate audio descriptions for movies and TV series, achieving state-of-the-art results without fine-tuning.

Contribution

It develops a novel two-stage prompting approach and creates a new TV audio description dataset, enabling effective zero-shot AD generation.

Findings

Achieves state-of-the-art CRITIC scores in AD generation

Demonstrates successful character referencing without fine-tuning

Outperforms some fine-tuned models in zero-shot setting

Abstract

Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs), and develop visual and text prompting strategies for this task. Our contributions are three-fold: (i) We demonstrate that a VLM can successfully name and refer to characters if directly prompted with character information through visual indications without requiring any fine-tuning; (ii) A two-stage process is developed to generate ADs, with the first stage asking the VLM to comprehensively describe the video, followed by a second stage utilising a LLM to summarise dense textual information into one succinct AD sentence; (iii) A new dataset for TV audio description is formulated. Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.