Robotic Classification of Divers' Swimming States using Visual Pose Keypoints as IMUs

TL;DR

This paper presents a hybrid computer vision approach that uses 3D human pose keypoints to classify divers' swimming states, improving underwater activity recognition and diver safety monitoring without relying on traditional IMUs.

Contribution

It introduces a novel method to generate pseudo-IMU data from visual pose keypoints, enabling effective underwater activity classification without wireless signal issues.

Findings

Successfully classified diver states using visual pose data

Detected anomalous behaviors indicating medical emergencies

Validated system onboard an AUV in simulated scenarios

Abstract

Traditional human activity recognition uses either direct image analysis or data from wearable inertial measurement units (IMUs), but can be ineffective in challenging underwater environments. We introduce a novel hybrid approach that bridges this gap to monitor scuba diver safety. Our method leverages computer vision to generate high-fidelity motion data, effectively creating a ``pseudo-IMU'' from a stream of 3D human joint keypoints. This technique circumvents the critical problem of wireless signal attenuation in water, which plagues conventional diver-worn sensors communicating with an Autonomous Underwater Vehicle (AUV). We apply this system to the vital task of identifying anomalous scuba diver behavior that signals the onset of a medical emergency such as cardiac arrest -- a leading cause of scuba diving fatalities. By integrating our classifier onboard an AUV and conducting experiments with simulated distress scenarios, we demonstrate the utility and effectiveness of our method for advancing robotic monitoring and diver safety.