Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction

TL;DR

This paper proposes a method to improve reinforcement learning agents by integrating constraint-based guidance, enhancing safety, reliability, and training efficiency in complex environments.

Contribution

It introduces a framework combining deep RL with constraint augmentation models to ensure safe and reliable agent behavior without hindering exploration.

Findings

Constraint guidance improves safety and reliability.

Accelerated training observed with constraint integration.

Effective in complex environments like card games and grid worlds.

Abstract

Reinforcement Learning (RL) agents have great successes in solving tasks with large observation and action spaces from limited feedback. Still, training the agents is data-intensive and there are no guarantees that the learned behavior is safe and does not violate rules of the environment, which has limitations for the practical deployment in real-world scenarios. This paper discusses the engineering of reliable agents via the integration of deep RL with constraint-based augmentation models to guide the RL agent towards safe behavior. Within the constraints set, the RL agent is free to adapt and explore, such that its effectiveness to solve the given problem is not hindered. However, once the RL agent leaves the space defined by the constraints, the outside models can provide guidance to still work reliably. We discuss integration points for constraint guidance within the RL process and perform experiments on two case studies: a strictly constrained card game and a grid world environment with additional combinatorial subgoals. Our results show that constraint-guidance does both provide reliability improvements and safer behavior, as well as accelerated training.