STREAMS: An Assistive Multimodal AI Framework for Empowering Biosignal Based Robotic Controls

TL;DR

STREAMS is a novel deep reinforcement learning framework that enhances biosignal-based robotic control, producing smooth trajectories and improving task success rates in assistive robotics without pre-existing datasets.

Contribution

The paper introduces STREAMS, a self-training, multimodal deep reinforcement learning framework that significantly improves biosignal-based robotic control in real-time applications.

Findings

Achieved 98% success in simulation with dynamic targets.

Demonstrated 83% success rate in real-world user study.

Significantly improved trajectory stability and user satisfaction.

Abstract

End-effector based assistive robots face persistent challenges in generating smooth and robust trajectories when controlled by human's noisy and unreliable biosignals such as muscle activities and brainwaves. The produced endpoint trajectories are often jerky and imprecise to perform complex tasks such as stable robotic grasping. We propose STREAMS (Self-Training Robotic End-to-end Adaptive Multimodal Shared autonomy) as a novel framework leveraged deep reinforcement learning to tackle this challenge in biosignal based robotic control systems. STREAMS blends environmental information and synthetic user input into a Deep Q Learning Network (DQN) pipeline for an interactive end-to-end and self-training mechanism to produce smooth trajectories for the control of end-effector based robots. The proposed framework achieved a high-performance record of 98% in simulation with dynamic target estimation and acquisition without any pre-existing datasets. As a zero-shot sim-to-real user study with five participants controlling a physical robotic arm with noisy head movements, STREAMS (as an assistive mode) demonstrated significant improvements in trajectory stabilization, user satisfaction, and task performance reported as a success rate of 83% compared to manual mode which was 44% without any task support. STREAMS seeks to improve biosignal based assistive robotic controls by offering an interactive, end-to-end solution that stabilizes end-effector trajectories, enhancing task performance and accuracy.