Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

TL;DR

This paper introduces M2SE-VTTS, a novel multi-modal and multi-scale approach that leverages RGB and Depth images to improve spatial environment understanding for immersive visual text-to-speech synthesis, outperforming existing methods.

Contribution

The paper proposes a new multi-modal, multi-scale framework that integrates RGB and Depth data with local-global spatial modeling for enhanced environmental speech synthesis.

Findings

Outperforms advanced baselines in objective evaluations

Achieves superior subjective quality in speech synthesis

Effectively models local and global spatial interactions

Abstract

Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize the reverberant speech for the spoken content. The challenge of this task lies in understanding the spatial environment from the image. Many attempts have been made to extract global spatial visual information from the RGB space of an spatial image. However, local and depth image information are crucial for understanding the spatial environment, which previous works have ignored. To address the issues, we propose a novel multi-modal and multi-scale spatial environment understanding scheme to achieve immersive VTTS, termed M2SE-VTTS. The multi-modal aims to take both the RGB and Depth spaces of the spatial image to learn more comprehensive spatial information, and the multi-scale seeks to model the local and global spatial knowledge simultaneously. Specifically, we first split the RGB and Depth images into patches and adopt the Gemini-generated environment captions to guide the local spatial understanding. After that, the multi-modal and multi-scale features are integrated by the local-aware global spatial understanding. In this way, M2SE-VTTS effectively models the interactions between local and global spatial contexts in the multi-modal spatial environment. Objective and subjective evaluations suggest that our model outperforms the advanced baselines in environmental speech generation. The code and audio samples are available at: https://github.com/AI-S2-Lab/M2SE-VTTS.