Learning Control Policies for Imitating Human Gaits

TL;DR

This paper presents a reinforcement learning framework for learning to imitate human gaits using skeletal and musculoskeletal models, demonstrating the superiority of muscle-tendon actuation in producing smooth, efficient movements.

Contribution

It introduces a novel RL-based approach for gait imitation that compares skeletal and musculoskeletal models, highlighting the advantages of muscle-tendon actuation.

Findings

Muscle-tendon models outperform skeletal models in smoothness and efficiency.

Reinforcement learning effectively optimizes control actions for gait imitation.

Musculoskeletal models reduce the need for external regularizers.

Abstract

The work presented in this report introduces a framework aimed towards learning to imitate human gaits. Humans exhibit movements like walking, running, and jumping in the most efficient manner, which served as the source of motivation for this project. Skeletal and Musculoskeletal human models were considered for motions in the sagittal plane, and results from both were compared exhaustively. While skeletal models are driven with motor actuation, musculoskeletal models perform through muscle-tendon actuation. Model-free reinforcement learning algorithms were used to optimize inverse dynamics control actions to satisfy the objective of imitating a reference motion along with secondary objectives of minimizing effort in terms of power spent by motors and metabolic energy consumed by the muscles. On the one hand, the control actions for the motor actuated model is the target joint angles converted into joint torques through a Proportional-Differential controller. While on the other hand, the control actions for the muscle-tendon actuated model is the muscle excitations converted implicitly to muscle activations and then to muscle forces which apply moments on joints. Muscle-tendon actuated models were found to have superiority over motor actuation as they are inherently smooth due to muscle activation dynamics and don't need any external regularizers. Finally, a strategy that was used to obtain an optimal configuration of the significant decision variables in the framework was discussed. All the results and analysis are presented in an illustrative, qualitative, and quantitative manner. Supporting video links are provided in the Appendix.