A Wearable Multi-Modal Edge-Computing System for Real-Time Kitchen Activity Recognition

TL;DR

This paper introduces a multi-modal wearable edge computing system capable of real-time kitchen activity recognition, combining diverse sensors and microcontrollers to achieve high accuracy and efficiency locally, reducing latency and privacy risks.

Contribution

It presents a novel integrated wearable system with multi-modal sensors and microcontrollers for end-to-end real-time activity recognition on energy-efficient devices.

Findings

Achieves 87.83% accuracy in recognizing 15 kitchen activities.

Inference time of 25.26 ms on microcontroller.

Demonstrates reduced power consumption and fast inference compared to alternatives.

Abstract

In the human activity recognition research area, prior studies predominantly concentrate on leveraging advanced algorithms on public datasets to enhance recognition performance, little attention has been paid to executing real-time kitchen activity recognition on energy-efficient, cost-effective edge devices. Besides, the prevalent approach of segregating data collection and context extraction across different devices escalates power usage, latency, and user privacy risks, impeding widespread adoption. This work presents a multi-modal wearable edge computing system for human activity recognition in real-time. Integrating six different sensors, ranging from inertial measurement units (IMUs) to thermal cameras, and two different microcontrollers, this system achieves end-to-end activity recognition, from data capture to context extraction, locally. Evaluation in an unmodified realistic kitchen validates its efficacy in recognizing fifteen activities, including a null class. Employing a compact machine learning model (184.5 kbytes) yields an average accuracy of 87.83 \%, with model inference completed in 25.26 ms on the microcontroller. Comparative analysis with alternative microcontrollers showcases power consumption and inference speed performance, demonstrating the proposed system's viability.