Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations

TL;DR

This paper introduces two novel Hamilton-Jacobi-Bellman based value functions for dual-objective reinforcement learning, enabling explicit satisfaction of complex constraints without reward engineering, and demonstrates their effectiveness with a new PPO variant.

Contribution

It extends Hamilton-Jacobi RL to handle dual objectives with explicit Bellman forms, providing a tractable approach for constrained RL problems.

Findings

Outperforms baselines in success and safety metrics

Produces distinct behaviors from previous methods

Effective in safe-arrival and multi-target tasks

Abstract

Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem -- of achieving distinct reward and penalty thresholds -- and 2) the Reach-Reach (RR) problem -- of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DOHJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, outcompeting a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.