Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectives

TL;DR

This paper introduces a novel model-free reinforcement learning framework that optimizes absolute liveness specifications expressed as omega-regular languages using average-reward objectives, suitable for ongoing tasks without resets.

Contribution

It is the first to translate absolute liveness omega-regular specifications into average-reward objectives for model-free RL in unknown MDPs, supporting continuous interaction.

Findings

Outperforms discount-based methods in benchmarks

Guarantees convergence in unknown communicating MDPs

Supports on-the-fly environment reductions

Abstract

Recent advances in reinforcement learning (RL) have renewed interest in reward design for shaping agent behavior, but manually crafting reward functions is tedious and error-prone. A principled alternative is to specify behavioral requirements in a formal, unambiguous language and automatically compile them into learning objectives. $\omega$-regular languages are a natural fit, given their role in formal verification and synthesis. However, most existing $\omega$-regular RL approaches operate in an episodic, discounted setting with periodic resets, which is misaligned with $\omega$-regular semantics over infinite traces. For continuing tasks, where the agent interacts with the environment over a single uninterrupted lifetime, the average-reward criterion is more appropriate. We focus on absolute liveness specifications, a subclass of $\omega$-regular languages that cannot be violated by any finite prefix and thus aligns naturally with continuing interaction. We present the first model-free RL framework that translates absolute liveness specifications into average-reward objectives and enables learning in unknown communicating Markov decision processes (MDPs) without episodic resetting. We also introduce a reward structure for lexicographic multi-objective optimization: among policies that maximize the satisfaction probability of an absolute liveness specification, the agent maximizes an external average-reward objective. Our method guarantees convergence in unknown communicating MDPs and supports on-the-fly reductions that do not require full environment knowledge, enabling model-free learning. Experiments across several benchmarks show that the continuing, average-reward approach outperforms competing discount-based methods.