Risk-Aware Reinforcement Learning through Optimal Transport Theory

TL;DR

This paper introduces a novel risk-aware reinforcement learning framework that integrates Optimal Transport theory to balance reward maximization with risk constraints, providing a mathematically rigorous approach to risk management in uncertain environments.

Contribution

It pioneers the integration of Optimal Transport theory into reinforcement learning to explicitly incorporate risk considerations into the policy optimization process.

Findings

Theoretical formulation of risk-aware RL using OT distances.

Proven theorems linking risk distributions, value functions, and policies.

Demonstrated the effectiveness of the approach in balancing reward and risk.

Abstract

In the dynamic and uncertain environments where reinforcement learning (RL) operates, risk management becomes a crucial factor in ensuring reliable decision-making. Traditional RL approaches, while effective in reward optimization, often overlook the landscape of potential risks. In response, this paper pioneers the integration of Optimal Transport (OT) theory with RL to create a risk-aware framework. Our approach modifies the objective function, ensuring that the resulting policy not only maximizes expected rewards but also respects risk constraints dictated by OT distances between state visitation distributions and the desired risk profiles. By leveraging the mathematical precision of OT, we offer a formulation that elevates risk considerations alongside conventional RL objectives. Our contributions are substantiated with a series of theorems, mapping the relationships between risk distributions, optimal value functions, and policy behaviors. Through the lens of OT, this work illuminates a promising direction for RL, ensuring a balanced fusion of reward pursuit and risk awareness.