Single-stage intake gesture detection using CTC loss and extended prefix beam search

TL;DR

This paper introduces a novel single-stage method for detecting intake gestures directly from sensor data using CTC loss and extended prefix beam search, improving accuracy over traditional two-stage methods.

Contribution

It presents a new end-to-end approach for intake gesture detection that simplifies training and enhances detection performance compared to existing methods.

Findings

Achieved 1.9% to 6.2% higher F1 scores across datasets.

Demonstrated improved detection accuracy for both video and inertial sensors.

Enabled end-to-end training with relaxed timing label requirements.

Abstract

Accurate detection of individual intake gestures is a key step towards automatic dietary monitoring. Both inertial sensor data of wrist movements and video data depicting the upper body have been used for this purpose. The most advanced approaches to date use a two-stage approach, in which (i) frame-level intake probabilities are learned from the sensor data using a deep neural network, and then (ii) sparse intake events are detected by finding the maxima of the frame-level probabilities. In this study, we propose a single-stage approach which directly decodes the probabilities learned from sensor data into sparse intake detections. This is achieved by weakly supervised training using Connectionist Temporal Classification (CTC) loss, and decoding using a novel extended prefix beam search decoding algorithm. Benefits of this approach include (i) end-to-end training for detections, (ii) simplified timing requirements for intake gesture labels, and (iii) improved detection performance compared to existing approaches. Across two separate datasets, we achieve relative $F_1$ score improvements between 1.9% and 6.2% over the two-stage approach for intake detection and eating/drinking detection tasks, for both video and inertial sensors.