Bellman Value Decomposition for Task Logic in Safe Optimal Control

TL;DR

This paper introduces a novel Bellman value decomposition approach for complex high-dimensional tasks with safety and goal specifications, enabling improved automatic performance in safe optimal control.

Contribution

It proves a new Bellman value decomposition for tasks in temporal logic and proposes VDPPO, a neural network method leveraging this structure for better safety and liveness balance.

Findings

Significant performance improvements over baselines in simulated and hardware experiments.

Effective handling of complex, high-dimensional tasks with safety and goal constraints.

Automatic balancing of safety and liveness in control tasks.

Abstract

Real-world tasks involve nuanced combinations of goal and safety specifications. In high dimensions, the challenge is exacerbated: formal automata become cumbersome, and the combination of sparse rewards tends to require laborious tuning. In this work, we consider the innate structure of the Bellman Value as a means to naturally organize the problem for improved automatic performance. Namely, we prove the Bellman Value for a complex task defined in temporal logic can be decomposed into a graph of Bellman Values, connected by a set of well-known Bellman equations (BEs): the Reach-Avoid BE, the Avoid BE, and a novel type, the Reach-Avoid-Loop BE. To solve the Value and optimal policy, we propose VDPPO, which embeds the decomposed Value graph into a two-layer neural net, bootstrapping the implicit dependencies. We conduct a variety of simulated and hardware experiments to test our method on complex, high-dimensional tasks involving heterogeneous teams and nonlinear dynamics. Ultimately, we find this approach greatly improves performance over existing baselines, balancing safety and liveness automatically.