MetaSpeech: Speech Effects Switch Along with Environment for Metaverse

TL;DR

MetaSpeech is a novel voice conversion method that dynamically switches environmental effects in speech, enhancing user immersion in the Metaverse by accurately modeling and transferring environment-specific audio effects.

Contribution

The paper introduces MetaSpeech, a new environment effect conversion model using adversarial training to disentangle speech content, speaker identity, and environmental effects for immersive Metaverse experiences.

Findings

Successfully converts speech to target environment effects

Outperforms baseline voice conversion methods on LJSpeech dataset

Effectively disentangles environmental effects from speech content

Abstract

Metaverse expands the physical world to a new dimension, and the physical environment and Metaverse environment can be directly connected and entered. Voice is an indispensable communication medium in the real world and Metaverse. Fusion of the voice with environment effects is important for user immersion in Metaverse. In this paper, we proposed using the voice conversion based method for the conversion of target environment effect speech. The proposed method was named MetaSpeech, which introduces an environment effect module containing an effect extractor to extract the environment information and an effect encoder to encode the environment effect condition, in which gradient reversal layer was used for adversarial training to keep the speech content and speaker information while disentangling the environmental effects. From the experiment results on the public dataset of LJSpeech with four environment effects, the proposed model could complete the specific environment effect conversion and outperforms the baseline methods from the voice conversion task.