Simultaneous Speech and Eating Behavior Recognition Using Data Augmentation and Two-Stage Fine-Tuning

TL;DR

This paper introduces a new method to recognize both speech and eating behaviors at the same time, using data augmentation and two-stage fine-tuning to improve accuracy in health monitoring.

Contribution

The novel approach combines synthetic data augmentation with two-stage fine-tuning for simultaneous speech and eating behavior recognition.

Findings

The method achieves an F1 score of 0.918 for chewing detection and 0.926 for swallowing detection.

Speech recognition accuracy is maintained while improving eating behavior detection performance.

Abstract

Speaking and eating are essential components of health management. To enable the daily monitoring of these behaviors, systems capable of simultaneously recognizing speech and eating behaviors are required. However, due to the distinct acoustic and contextual characteristics of these two domains, achieving high-precision integrated recognition remains underexplored. In this study, we propose a method that combines data augmentation through synthetic data creation with a two-stage fine-tuning approach tailored to the complexity of domain adaptation. By concatenating speech and eating sounds of varying lengths and sequences, we generated training data that mimic real-world environments where speech and eating behaviors co-exist. Additionally, efficient model adaptation was achieved through two-stage fine-tuning of the self-supervised learning model. The experimental evaluations demonstrate that the proposed method maintains speech recognition accuracy while achieving high detection performance for eating behaviors, with an F1 score of 0.918 for chewing detection and 0.926 for swallowing detection. These results underscore the potential of using voice recognition technology for daily health monitoring.

Figures