Copilot-Assisted Second-Thought Framework for Brain-to-Robot Hand Motion Decoding

TL;DR

This paper introduces a hybrid CNN-attention model for decoding hand movements from EEG and EMG data, improving brain-to-robot control accuracy with a copilot framework that filters low-confidence predictions.

Contribution

It presents a novel CNN-attention hybrid model for multimodal EEG-EMG decoding and a copilot framework that enhances trajectory fidelity in brain-to-robot interfaces.

Findings

Achieved PCC of 0.9854, 0.9946, 0.9065 in within-subject tests for X, Y, Z axes.

EEG-EMG multimodal decoding significantly improves accuracy.

Copilot framework increases EEG-only decoding PCC to 0.93, filtering fewer than 20% of data.

Abstract

Motor kinematics prediction (MKP) from electroencephalography (EEG) is an important research area for developing movement-related brain-computer interfaces (BCIs). While traditional methods often rely on convolutional neural networks (CNNs) or recurrent neural networks (RNNs), Transformer-based models have shown strong ability in modeling long sequential EEG data. In this study, we propose a CNN-attention hybrid model for decoding hand kinematics from EEG during grasp-and-lift tasks, achieving strong performance in within-subject experiments. We further extend this approach to EEG-EMG multimodal decoding, which yields substantially improved results. Within-subject tests achieve PCC values of 0.9854, 0.9946, and 0.9065 for the X, Y, and Z axes, respectively, computed on the midpoint trajectory between the thumb and index finger, while cross-subject tests result in 0.9643, 0.9795, and 0.5852. The decoded trajectories from both modalities are then used to control a Franka Panda robotic arm in a MuJoCo simulation. To enhance trajectory fidelity, we introduce a copilot framework that filters low-confidence decoded points using a motion-state-aware critic within a finite-state machine. This post-processing step improves the overall within-subject PCC of EEG-only decoding to 0.93 while excluding fewer than 20% of the data points.