Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes

TL;DR

This paper introduces RED reinforcement learning, a novel framework for solving multiple subtasks in average-reward MDPs, enabling risk-aware decision-making like CVaR optimization in an online setting.

Contribution

The work presents the first effective RL algorithms for average-reward MDPs that handle multiple subtasks and incorporate risk measures such as CVaR without complex optimization schemes.

Findings

Proposed RED algorithms are proven to converge in tabular cases.

Demonstrated online CVaR optimization in average-reward MDPs.

Unified framework for multiple subtasks in average-reward RL.

Abstract

Average-reward Markov decision processes (MDPs) provide a foundational framework for sequential decision-making under uncertainty. However, average-reward MDPs have remained largely unexplored in reinforcement learning (RL) settings, with the majority of RL-based efforts having been allocated to discounted MDPs. In this work, we study a unique structural property of average-reward MDPs and utilize it to introduce Reward-Extended Differential (or RED) reinforcement learning: a novel RL framework that can be used to effectively and efficiently solve various learning objectives, or subtasks, simultaneously in the average-reward setting. We introduce a family of RED learning algorithms for prediction and control, including proven-convergent algorithms for the tabular case. We then showcase the power of these algorithms by demonstrating how they can be used to learn a policy that optimizes, for the first time, the well-known conditional value-at-risk (CVaR) risk measure in a fully-online manner, without the use of an explicit bi-level optimization scheme or an augmented state-space.