Visualizations of Complex Sequences of Family-Infant Vocalizations Using Bag-of-Audio-Words Approach Based on Wav2vec 2.0 Features

TL;DR

This paper introduces a novel visualization method for analyzing family-infant vocalization interactions using wav2vec 2.0 features and a bag-of-audio-words approach, aiding early detection of developmental issues.

Contribution

It presents a new visualization technique leveraging wav2vec 2.0 and bag-of-audio-words to analyze infant vocalizations with improved labeling and interpretability.

Findings

High-quality visualizations capture key vocalization interaction types

Method distinguishes between different family-infant vocalization patterns

Approach applicable to both labeled and unlabeled audio data

Abstract

In the U.S., approximately 15-17% of children 2-8 years of age are estimated to have at least one diagnosed mental, behavioral or developmental disorder. However, such disorders often go undiagnosed, and the ability to evaluate and treat disorders in the first years of life is limited. To analyze infant developmental changes, previous studies have shown advanced ML models excel at classifying infant and/or parent vocalizations collected using cell phone, video, or audio-only recording device like LENA. In this study, we pilot test the audio component of a new infant wearable multi-modal device that we have developed called LittleBeats (LB). LB audio pipeline is advanced in that it provides reliable labels for both speaker diarization and vocalization classification tasks, compared with other platforms that only record audio and/or provide speaker diarization labels. We leverage wav2vec 2.0 to obtain superior and more nuanced results with the LB family audio stream. We use a bag-of-audio-words method with wav2vec 2.0 features to create high-level visualizations to understand family-infant vocalization interactions. We demonstrate that our high-quality visualizations capture major types of family vocalization interactions, in categories indicative of mental, behavioral, and developmental health, for both labeled and unlabeled LB audio.