VideoA11y: Method and Dataset for Accessible Video Description

TL;DR

VideoA11y introduces a new method and a large dataset for generating high-quality, accessible video descriptions tailored for blind and low vision users, leveraging multimodal large language models and guidelines.

Contribution

It presents a novel approach combining MLLMs and accessibility guidelines, and curates the largest dataset for BLV video descriptions, improving description quality and user satisfaction.

Findings

VideoA11y descriptions outperform novice annotations.

Descriptions are comparable to trained human annotations.

Fine-tuned MLLMs on VideoA11y-40K produce high-quality outputs.

Abstract

Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall short due to limitations in the quality of human annotations within training datasets, resulting in descriptions that do not fully meet BLV users' needs. To address this gap, we introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals. Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users. Rigorous experiments across 15 video categories, involving 347 sighted participants, 40 BLV participants, and seven professional describers, showed that VideoA11y descriptions outperform novice human annotations and are comparable to trained human annotations in clarity, accuracy, objectivity, descriptiveness, and user satisfaction. We evaluated models on VideoA11y-40K using both standard and custom metrics, demonstrating that MLLMs fine-tuned on this dataset produce high-quality accessible descriptions. Code and dataset are available at https://people-robots.github.io/VideoA11y.