Progress Constraints for Reinforcement Learning in Behavior Trees

TL;DR

This paper introduces progress constraints to improve the integration of Behavior Trees and Reinforcement Learning, enhancing performance, safety, and efficiency in decision-making tasks.

Contribution

It proposes a novel progress constraints mechanism that uses feasibility estimators to ensure safe and effective RL within BTs, based on theoretical convergence results.

Findings

Enhanced performance in simulated environments

Improved sample efficiency over prior methods

Better constraint satisfaction and safety

Abstract

Behavior Trees (BTs) provide a structured and reactive framework for decision-making, commonly used to switch between sub-controllers based on environmental conditions. Reinforcement Learning (RL), on the other hand, can learn near-optimal controllers but sometimes struggles with sparse rewards, safe exploration, and long-horizon credit assignment. Combining BTs with RL has the potential for mutual benefit: a BT design encodes structured domain knowledge that can simplify RL training, while RL enables automatic learning of the controllers within BTs. However, naive integration of BTs and RL can lead to some controllers counteracting other controllers, possibly undoing previously achieved subgoals, thereby degrading the overall performance. To address this, we propose progress constraints, a novel mechanism where feasibility estimators constrain the allowed action set based on theoretical BT convergence results. Empirical evaluations in a 2D proof-of-concept and a high-fidelity warehouse environment demonstrate improved performance, sample efficiency, and constraint satisfaction, compared to prior methods of BT-RL integration.