Laugh Betrays You? Learning Robust Speaker Representation From Speech Containing Non-Verbal Fragments

TL;DR

This paper investigates speaker verification using speech with non-verbal laughter segments, proposing a novel Laughter-Splicing Network that improves performance in laughter-inclusive scenarios.

Contribution

It introduces a new approach with Laughter-Splicing Network for speaker verification involving laughter, enhancing robustness in non-verbal speech conditions.

Findings

20% relative improvement on Laughter-Laughter trials

22% relative improvement on Speech-Laughter trials

Maintains performance on neutral speech datasets

Abstract

The success of automatic speaker verification shows that discriminative speaker representations can be extracted from neutral speech. However, as a kind of non-verbal voice, laughter should also carry speaker information intuitively. Thus, this paper focuses on exploring speaker verification about utterances containing non-verbal laughter segments. We collect a set of clips with laughter components by conducting a laughter detection script on VoxCeleb and part of the CN-Celeb dataset. To further filter untrusted clips, probability scores are calculated by our binary laughter detection classifier, which is pre-trained by pure laughter and neutral speech. After that, based on the clips whose scores are over the threshold, we construct trials under two different evaluation scenarios: Laughter-Laughter (LL) and Speech-Laughter (SL). Then a novel method called Laughter-Splicing based Network (LSN) is proposed, which can significantly boost performance in both scenarios and maintain the performance on the neutral speech, such as the VoxCeleb1 test set. Specifically, our system achieves relative 20% and 22% improvement on Laughter-Laughter and Speech-Laughter trials, respectively. The meta-data and sample clips have been released at https://github.com/nevermoreLin/Laugh_LSN.