Reinforcement Learning for High-dimensional Continuous Control in Biomechanics: An Intro to ArtiSynth-RL

TL;DR

This paper introduces ArtiSynth, a 3D biomechanical simulation platform extended for reinforcement learning, and demonstrates its use in training deep RL policies for physiologically accurate motor control in high-dimensional continuous spaces.

Contribution

The work presents ArtiSynth-RL, a novel integration of biomechanical simulation with reinforcement learning, enabling training of control policies in complex, physiologically accurate models.

Findings

Deep RL policies can effectively control biomechanical models.

ArtiSynth-RL outperforms traditional inverse dynamic optimization in stability and energy efficiency.

The platform facilitates comprehensive evaluation of control strategies in realistic simulations.

Abstract

Neural control is an exciting mystery which we instinctively master. Yet, researchers have a hard time explaining the motor control trajectories. Physiologically accurate biomechanical simulations can, to some extent, mimic live subjects and help us form evidence-based hypotheses. In these simulated environments, muscle excitations are typically calculated through inverse dynamic optimizations which do not possess a closed-form solution. Thus, computationally expensive, and occasionally unstable, iterative numerical solvers are the only widely utilized solution. In this work, we introduce ArtiSynth, a 3D modeling platform that supports the combined simulation of multi-body and finite element models, and extended to support reinforcement learning (RL) training. we further use ArtiSynth to investigate whether a deep RL policy can be trained to drive the motor control of a physiologically accurate biomechanical model in a large continuous action space. We run a comprehensive evaluation of its performance and compare the results with the forward dynamics assisted tracking with a quadratic objective function. We assess the two approaches in terms of correctness, stability, energy-efficiency, and temporal consistency.