Actor-Critic for Linearly-Solvable Continuous MDP with Partially Known Dynamics

TL;DR

This paper introduces an actor-critic reinforcement learning method for linearly-solvable continuous MDPs that learns with partial knowledge of system dynamics without active exploration, improving policy performance.

Contribution

It presents a novel actor-critic RL approach for L-MDPs that does not require modeling uncontrolled dynamics or transition noise, only control dynamics, suitable for real-world applications.

Findings

Improved learning efficiency demonstrated on synthetic problems

Effective policy performance in simulated traffic data

No need for active exploration or full system modeling

Abstract

In many robotic applications, some aspects of the system dynamics can be modeled accurately while others are difficult to obtain or model. We present a novel reinforcement learning (RL) method for continuous state and action spaces that learns with partial knowledge of the system and without active exploration. It solves linearly-solvable Markov decision processes (L-MDPs), which are well suited for continuous state and action spaces, based on an actor-critic architecture. Compared to previous RL methods for L-MDPs and path integral methods which are model based, the actor-critic learning does not need a model of the uncontrolled dynamics and, importantly, transition noise levels; however, it requires knowing the control dynamics for the problem. We evaluate our method on two synthetic test problems, and one real-world problem in simulation and using real traffic data. Our experiments demonstrate improved learning and policy performance.