ViDscribe: Multimodal AI for Customizing Audio Description and Question Answering in Online Videos

TL;DR

ViDscribe is a web platform that uses multimodal AI to generate customizable audio descriptions and question answering for online videos, enhancing accessibility for blind and low vision users through personalization and interaction.

Contribution

The paper introduces ViDscribe, a novel platform combining AI-generated customizable audio descriptions with a conversational VQA interface, tested in real-world settings with BLV users.

Findings

Customized ADs improve user engagement and satisfaction.

Users prefer personalized descriptions over default ones.

Interactive features increase immersion for BLV viewers.

Abstract

Advances in multimodal large language models enable automatic video narration and question answering (VQA), offering scalable alternatives to labor-intensive, human-authored audio descriptions (ADs) for blind and low vision (BLV) viewers. However, prior AI-driven AD systems rarely adapt to the diverse needs and preferences of BLV individuals across videos and are typically evaluated in controlled, single-session settings. We present ViDscribe, a web-based platform that integrates AI-generated ADs with six types of user customizations and a conversational VQA interface for YouTube videos. Through a longitudinal, in-the-wild study with eight BLV participants, we examine how users engage with customization and VQA features over time. Our results show sustained engagement with both features and that customized ADs improve effectiveness, enjoyment, and immersion compared to default ADs, highlighting the value of personalized, interactive video access for BLV users.