Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learning

TL;DR

This paper introduces a reinforcement learning algorithm that jointly trains a trajectory planner and a tracking controller in layered control systems, improving coordination and interpretability.

Contribution

It presents a novel actor-critic RL approach with a dual network for coordinating planning and tracking layers, including theoretical convergence proof in LQR and empirical validation on nonlinear systems.

Findings

Converges to optimal dual network in LQR setting

Effective coordination between planning and tracking layers

Validated on nonlinear unicycle model simulations

Abstract

We propose a reinforcement learning (RL)-based algorithm to jointly train (1) a trajectory planner and (2) a tracking controller in a layered control architecture. Our algorithm arises naturally from a rewrite of the underlying optimal control problem that lends itself to an actor-critic learning approach. By explicitly learning a \textit{dual} network to coordinate the interaction between the planning and tracking layers, we demonstrate the ability to achieve an effective consensus between the two components, leading to an interpretable policy. We theoretically prove that our algorithm converges to the optimal dual network in the Linear Quadratic Regulator (LQR) setting and empirically validate its applicability to nonlinear systems through simulation experiments on a unicycle model.