Model-Based Actor-Critic with Chance Constraint for Stochastic System

TL;DR

This paper introduces a model-based actor-critic algorithm that efficiently learns safe policies in stochastic systems by directly solving chance constraints, demonstrating faster convergence and higher efficiency than existing methods.

Contribution

The proposed CCAC algorithm directly addresses chance constraints in RL, improving convergence speed and safety performance over prior conservative or slow methods.

Findings

CCAC achieves five times faster convergence than previous RL methods.

CCAC maintains safety while improving performance in stochastic car-following tasks.

CCAC is 100 times more computationally efficient than traditional safety techniques.

Abstract

Safety is essential for reinforcement learning (RL) applied in real-world situations. Chance constraints are suitable to represent the safety requirements in stochastic systems. Previous chance-constrained RL methods usually have a low convergence rate, or only learn a conservative policy. In this paper, we propose a model-based chance constrained actor-critic (CCAC) algorithm which can efficiently learn a safe and non-conservative policy. Different from existing methods that optimize a conservative lower bound, CCAC directly solves the original chance constrained problems, where the objective function and safe probability is simultaneously optimized with adaptive weights. In order to improve the convergence rate, CCAC utilizes the gradient of dynamic model to accelerate policy optimization. The effectiveness of CCAC is demonstrated by a stochastic car-following task. Experiments indicate that compared with previous RL methods, CCAC improves the performance while guaranteeing safety, with a five times faster convergence rate. It also has 100 times higher online computation efficiency than traditional safety techniques such as stochastic model predictive control.