Reinforcement Learning with $\omega$-Regular Objectives and Constraints

TL;DR

This paper introduces a model-based reinforcement learning approach that optimizes complex temporal and safety-critical objectives expressed as $oldsymbol{ ext{omega}}$-regular properties, addressing limitations of scalar rewards and safety-performance trade-offs.

Contribution

It develops a linear programming-based RL algorithm for $oldsymbol{ ext{omega}}$-regular objectives with explicit constraints, ensuring safety and performance are managed separately.

Findings

Algorithm converges to policies maximizing $oldsymbol{ ext{omega}}$-regular satisfaction probabilities.

Provides a translation to constrained limit-average problems with guarantees.

Addresses safety-performance trade-offs in complex goal specifications.

Abstract

Reinforcement learning (RL) commonly relies on scalar rewards with limited ability to express temporal, conditional, or safety-critical goals, and can lead to reward hacking. Temporal logic expressible via the more general class of $\omega$-regular objectives addresses this by precisely specifying rich behavioural properties. Even still, measuring performance by a single scalar (be it reward or satisfaction probability) masks safety-performance trade-offs that arise in settings with a tolerable level of risk. We address both limitations simultaneously by combining $\omega$-regular objectives with explicit constraints, allowing safety requirements and optimisation targets to be treated separately. We develop a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an $\omega$-regular objective while also adhering to $\omega$-regular constraints within specified thresholds. Furthermore, we establish a translation to constrained limit-average problems with optimality-preserving guarantees.