Probabilistic Differentiable Filters Enable Ubiquitous Robot Control with Smartwatches

TL;DR

This paper introduces a probabilistic differentiable filter approach using smartwatches and smartphones for accurate, cost-effective human pose estimation to enable ubiquitous robot control and collaboration.

Contribution

It presents the Differentiable Ensemble Kalman Filter (DEnKF) for robot control using only IMUs from smart devices, a novel method for sparse, accurate human pose estimation.

Findings

Achieves accurate human pose estimation with IMUs from smart devices.

Enables versatile robot control in human-robot handover tasks.

Demonstrates cost-effective and ubiquitous robot control capabilities.

Abstract

Ubiquitous robot control and human-robot collaboration using smart devices poses a challenging problem primarily due to strict accuracy requirements and sparse information. This paper presents a novel approach that incorporates a probabilistic differentiable filter, specifically the Differentiable Ensemble Kalman Filter (DEnKF), to facilitate robot control solely using Inertial Measurement Units (IMUs) from a smartwatch and a smartphone. The implemented system is cost-effective and achieves accurate estimation of the human pose state. Experiment results from human-robot handover tasks underscore that smart devices allow versatile and ubiquitous robot control. The code for this paper is available at https://github.com/ir-lab/DEnKF and https://github.com/wearable-motion-capture.