Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audio

TL;DR

This paper demonstrates that unsupervised pretraining of Wav2vec 2.0 on large-scale unlabeled family audio significantly improves infant and parent speaker diarization and vocalization classification in real-world home recordings.

Contribution

It introduces a new approach using Wav2vec 2.0 pretrained on unlabeled family audio, outperforming models pretrained on general speech datasets for family audio analysis.

Findings

W2V2 pretrained on 1k-hour family recordings outperforms LibriSpeech pretrained models.

Additional unlabeled and labeled data further improve performance.

Data augmentation techniques yield 12% relative gain in speaker diarization.

Abstract

To perform automatic family audio analysis, past studies have collected recordings using phone, video, or audio-only recording devices like LENA, investigated supervised learning methods, and used or fine-tuned general-purpose embeddings learned from large pretrained models. In this study, we advance the audio component of a new infant wearable multi-modal device called LittleBeats (LB) by learning family audio representation via wav2vec 2.0 (W2V2) pertaining. We show given a limited number of labeled LB home recordings, W2V2 pretrained using 1k-hour of unlabeled home recordings outperforms oracle W2V2 pretrained on 960-hour unlabeled LibriSpeech in terms of parent/infant speaker diarization (SD) and vocalization classifications (VC) at home. Extra relevant external unlabeled and labeled data further benefit W2V2 pretraining and fine-tuning. With SpecAug and environmental speech corruptions, we obtain 12% relative gain on SD and moderate boost on VC. Code and model weights are available.