MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on Eyeglasses

TL;DR

MunchSonic is an innovative acoustic sensing system embedded in eyeglasses that accurately detects detailed dietary actions like eating, drinking, and talking using ultrasonic signals and deep learning, enabling real-time monitoring of eating behaviors.

Contribution

This work introduces MunchSonic, the first active ultrasonic sensing system integrated into eyeglasses for fine-grained dietary action detection using deep learning.

Findings

Achieved 93.5% macro F1-score in action classification

Effectively tracks eating episodes and food intake frequency

Demonstrated robustness in unconstrained real-world settings

Abstract

We introduce MunchSonic, an AI-powered active acoustic sensing system integrated into eyeglasses to track fine-grained dietary actions. MunchSonic emits inaudible ultrasonic waves from the eyeglass frame, with the reflected signals capturing detailed positions and movements of body parts, including the mouth, jaw, arms, and hands involved in eating. These signals are processed by a deep learning pipeline to classify six actions: hand-to-mouth movements for food intake, chewing, drinking, talking, face-hand touching, and other activities (null). In an unconstrained study with 12 participants, MunchSonic achieved a 93.5% macro F1-score in a user-independent evaluation with a 2-second resolution in tracking these actions, also demonstrating its effectiveness in tracking eating episodes and food intake frequency within those episodes.