V-CASS: Vision-context-aware Expressive Speech Synthesis for Enhancing User Understanding of Videos

TL;DR

V-CASS is a novel speech synthesis method that uses visual context to generate expressive speech, improving user understanding and engagement with videos, especially for visually impaired users.

Contribution

This paper introduces V-CASS, a vision-context-aware speech synthesis approach that aligns speech with visual cues to enhance video comprehension and accessibility.

Findings

V-CASS improves emotional resonance and user engagement.

74.68% of users preferred V-CASS over baseline methods.

V-CASS aids blind and low-vision users in navigating web videos.

Abstract

Automatic video commentary systems are widely used on multimedia social media platforms to extract factual information about video content. However, current systems may overlook essential para-linguistic cues, including emotion and attitude, which are critical for fully conveying the meaning of visual content. The absence of these cues can limit user understanding or, in some cases, distort the video's original intent. Expressive speech effectively conveys these cues and enhances the user's comprehension of videos. Building on these insights, this paper explores the usage of vision-context-aware expressive speech in enhancing users' understanding of videos in video commentary systems. Firstly, our formatting study indicates that semantic-only speech can lead to ambiguity, and misaligned emotions between speech and visuals may distort content interpretation. To address this, we propose a method called vision-context-aware speech synthesis (V-CASS). It analyzes para-linguistic cues from visuals using a vision-language model and leverages a knowledge-infused language model to guide the expressive speech model in generating context-aligned speech. User studies show that V-CASS enhances emotional and attitudinal resonance, as well as user audio-visual understanding and engagement, with 74.68% of participants preferring the system. Finally, we explore the potential of our method in helping blind and low-vision users navigate web videos, improving universal accessibility.