Reinforcement Learning based Constrained Optimal Control: an Interpretable Reward Design

TL;DR

This paper introduces an interpretable reward design framework for reinforcement learning in constrained optimal control, ensuring constraint satisfaction and improved control performance through theoretically justified weights and curriculum learning.

Contribution

It proposes a reward design method with bounds on component weights, integrating curriculum learning to enhance RL convergence in constrained control tasks.

Findings

Significantly improves constraint satisfaction in RL control tasks.

Enhances control cost optimization compared to baseline methods.

Demonstrates effectiveness in multi-agent environments.

Abstract

This paper presents an interpretable reward design framework for reinforcement learning based constrained optimal control problems with state and terminal constraints. The problem is formalized within a standard partially observable Markov decision process framework. The reward function is constructed from four weighted components: a terminal constraint reward, a guidance reward, a penalty for state constraint violations, and a cost reduction incentive reward. A theoretically justified reward design is then presented, which establishes bounds on the weights of the components. This approach ensures that constraints are satisfied and objectives are optimized while mitigating numerical instability. Acknowledging the importance of prior knowledge in reward design, we sequentially solve two subproblems, using each solution to inform the reward design for the subsequent problem. Subsequently, we integrate reinforcement learning with curriculum learning, utilizing policies derived from simpler subproblems to assist in tackling more complex challenges, thereby facilitating convergence. The framework is evaluated against original and randomly weighted reward designs in a multi-agent particle environment. Experimental results demonstrate that the proposed approach significantly enhances satisfaction of terminal and state constraints and optimization of control cost.