Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling

TL;DR

This paper introduces a deep reinforcement learning approach integrated with neuromechanical modeling for real-time adaptive neurostimulation in FES cycling, significantly improving control accuracy and performance over traditional methods.

Contribution

The novel integration of personalized neuromechanical models into RL enables efficient training and effective real-time control of neurostimulation for FES cycling.

Findings

RL outperforms PID and Fuzzy Logic controllers in accuracy

System successfully maintains high cadence during fatigue

Achieved Silver medal at Cybathlon 2020 FES discipline

Abstract

Functional Electrical Stimulation (FES) can restore motion to a paralysed person's muscles. Yet, control stimulating many muscles to restore the practical function of entire limbs is an unsolved problem. Current neurostimulation engineering still relies on 20th Century control approaches and correspondingly shows only modest results that require daily tinkering to operate at all. Here, we present our state of the art Deep Reinforcement Learning (RL) developed for real time adaptive neurostimulation of paralysed legs for FES cycling. Core to our approach is the integration of a personalised neuromechanical component into our reinforcement learning framework that allows us to train the model efficiently without demanding extended training sessions with the patient and working out of the box. Our neuromechanical component includes merges musculoskeletal models of muscle and or tendon function and a multistate model of muscle fatigue, to render the neurostimulation responsive to a paraplegic's cyclist instantaneous muscle capacity. Our RL approach outperforms PID and Fuzzy Logic controllers in accuracy and performance. Crucially, our system learned to stimulate a cyclist's legs from ramping up speed at the start to maintaining a high cadence in steady state racing as the muscles fatigue. A part of our RL neurostimulation system has been successfully deployed at the Cybathlon 2020 bionic Olympics in the FES discipline with our paraplegic cyclist winning the Silver medal among 9 competing teams.