MPC-Net: A First Principles Guided Policy Search

TL;DR

MPC-Net introduces a novel imitation learning method guided by optimal control principles, enabling efficient learning of control policies that satisfy constraints and adapt to multimodal behaviors in robotic systems.

Contribution

The paper proposes a control policy learning approach using a loss function based on the control Hamiltonian, directly encoding optimality and constraints, with a mixture-of-expert neural network for quadrupedal robot control.

Findings

Successfully stabilizes multiple gaits on a real robot

Requires less than 10 minutes of demonstration data

Achieves improved constraint satisfaction

Abstract

We present an Imitation Learning approach for the control of dynamical systems with a known model. Our policy search method is guided by solutions from MPC. Typical policy search methods of this kind minimize a distance metric between the guiding demonstrations and the learned policy. Our loss function, however, corresponds to the minimization of the control Hamiltonian, which derives from the principle of optimality. Therefore, our algorithm directly attempts to solve the optimality conditions with a parameterized class of control laws. Additionally, the proposed loss function explicitly encodes the constraints of the optimal control problem and we provide numerical evidence that its minimization achieves improved constraint satisfaction. We train a mixture-of-expert neural network architecture for controlling a quadrupedal robot and show that this policy structure is well suited for such multimodal systems. The learned policy can successfully stabilize different gaits on the real walking robot from less than 10 min of demonstration data.