A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder

TL;DR

This paper presents an LSTM-VAE-based multimodal anomaly detection method for robot-assisted feeding, demonstrating superior performance in identifying diverse anomalies using fused sensory data.

Contribution

The paper introduces a novel LSTM-VAE model for multimodal sensor fusion and anomaly detection in assistive robotics, with improved accuracy over existing methods.

Findings

Achieved higher AUC (0.8710) than baseline detectors.

Effective multimodal fusion demonstrated with raw sensory signals.

Detector successfully identified 12 types of anomalies.

Abstract

The detection of anomalous executions is valuable for reducing potential hazards in assistive manipulation. Multimodal sensory signals can be helpful for detecting a wide range of anomalies. However, the fusion of high-dimensional and heterogeneous modalities is a challenging problem. We introduce a long short-term memory based variational autoencoder (LSTM-VAE) that fuses signals and reconstructs their expected distribution. We also introduce an LSTM-VAE-based detector using a reconstruction-based anomaly score and a state-based threshold. For evaluations with 1,555 robot-assisted feeding executions including 12 representative types of anomalies, our detector had a higher area under the receiver operating characteristic curve (AUC) of 0.8710 than 5 other baseline detectors from the literature. We also show the multimodal fusion through the LSTM-VAE is effective by comparing our detector with 17 raw sensory signals versus 4 hand-engineered features.