NowYouSee Me: Context-Aware Automatic Audio Description

TL;DR

This paper introduces CA3D, a novel end-to-end system for automatic, context-aware audio description that accurately localizes visual events in cinematic content using only visual cues, enhancing accessibility for visually impaired audiences.

Contribution

The paper presents CA3D, the first unified system that detects and generates audio descriptions solely from visual information, improving automation and accuracy over previous metadata-dependent methods.

Findings

Achieves state-of-the-art performance in AD event detection

Improves script generation metrics significantly

Demonstrates effectiveness across diverse cinematic content

Abstract

Audio Description (AD) plays a pivotal role as an application system aimed at guaranteeing accessibility in multimedia content, which provides additional narrations at suitable intervals to describe visual elements, catering specifically to the needs of visually impaired audiences. In this paper, we introduce $\mathrm{CA^3D}$, the pioneering unified Context-Aware Automatic Audio Description system that provides AD event scripts with precise locations in the long cinematic content. Specifically, $\mathrm{CA^3D}$ system consists of: 1) a Temporal Feature Enhancement Module to efficiently capture longer term dependencies, 2) an anchor-based AD event detector with feature suppression module that localizes the AD events and extracts discriminative feature for AD generation, and 3) a self-refinement module that leverages the generated output to tweak AD event boundaries from coarse to fine. Unlike conventional methods which rely on metadata and ground truth AD timestamp for AD detection and generation tasks, the proposed $\mathrm{CA^3D}$ is the first end-to-end trainable system that only uses visual cue. Extensive experiments demonstrate that the proposed $\mathrm{CA^3D}$ improves existing architectures for both AD event detection and script generation metrics, establishing the new state-of-the-art performances in the AD automation.