Reinforcement Learning of CPG-regulated Locomotion Controller for a Soft Snake Robot

TL;DR

This paper presents a bio-inspired reinforcement learning approach combined with a central pattern generator to achieve natural, smooth, and adaptable locomotion control for a soft snake robot, validated through simulation and real-world experiments.

Contribution

It introduces a novel integration of RL with a CPG for soft robot control, enhancing maneuverability and natural movement patterns, which is a significant advancement over existing methods.

Findings

The combined RL and CPG controller improves locomotion stability.

The approach achieves effective goal-tracking in simulation and real-world tests.

Compared to pure RL methods, the bio-inspired design offers better naturalness and robustness.

Abstract

Intelligent control of soft robots is challenging due to the nonlinear and difficult-to-model dynamics. One promising model-free approach for soft robot control is reinforcement learning (RL). However, model-free RL methods tend to be computationally expensive and data-inefficient and may not yield natural and smooth locomotion patterns for soft robots. In this work, we develop a bio-inspired design of a learning-based goal-tracking controller for a soft snake robot. The controller is composed of two modules: An RL module for learning goal-tracking behaviors given the unmodeled and stochastic dynamics of the robot, and a central pattern generator (CPG) with the Matsuoka oscillators for generating stable and diverse locomotion patterns. We theoretically investigate the maneuverability of Matsuoka CPG's oscillation bias, frequency, and amplitude for steering control, velocity control, and sim-to-real adaptation of the soft snake robot. Based on this analysis, we proposed a composition of RL and CPG modules such that the RL module regulates the tonic inputs to the CPG system given state feedback from the robot, and the output of the CPG module is then transformed into pressure inputs to pneumatic actuators of the soft snake robot. This design allows the RL agent to naturally learn to entrain the desired locomotion patterns determined by the CPG maneuverability. We validated the optimality and robustness of the control design in both simulation and real experiments, and performed extensive comparisons with state-of-art RL methods to demonstrate the benefit of our bio-inspired control design.