AutoAD III: The Prequel -- Back to the Pixels

TL;DR

This paper introduces new datasets, a novel model architecture, and evaluation metrics for generating detailed audio descriptions of movies, significantly advancing the state of the art in visual understanding and language generation for this task.

Contribution

It presents two methods for creating aligned AD datasets, a Q-former-based model architecture utilizing frozen encoders and large language models, and new evaluation metrics tailored for AD quality assessment.

Findings

Improved AD generation performance over previous methods

Public release of new aligned AD datasets

Development of evaluation metrics aligned with human judgment

Abstract

Generating Audio Description (AD) for movies is a challenging task that requires fine-grained visual understanding and an awareness of the characters and their names. Currently, visual language models for AD generation are limited by a lack of suitable training data, and also their evaluation is hampered by using performance measures not specialized to the AD domain. In this paper, we make three contributions: (i) We propose two approaches for constructing AD datasets with aligned video data, and build training and evaluation datasets using these. These datasets will be publicly released; (ii) We develop a Q-former-based architecture which ingests raw video and generates AD, using frozen pre-trained visual encoders and large language models; and (iii) We provide new evaluation metrics to benchmark AD quality that are well-matched to human performance. Taken together, we improve the state of the art on AD generation.