Policy Gradients for Cumulative Prospect Theory in Reinforcement Learning

TL;DR

This paper develops a policy gradient method for reinforcement learning with Cumulative Prospect Theory objectives, incorporating behavioral economic insights and providing convergence guarantees.

Contribution

It generalizes the policy gradient theorem to CPT, introduces a Monte Carlo estimator, and proves convergence for CPT-based RL algorithms.

Findings

The algorithm converges to stationary points of the CPT objective.

Simulations demonstrate qualitative behaviors of CPT in RL.

The method outperforms existing zeroth-order approaches.

Abstract

We derive a policy gradient theorem for Cumulative Prospect Theory (CPT) objectives in finite-horizon Reinforcement Learning (RL), generalizing the standard policy gradient theorem and encompassing distortion-based risk objectives as special cases. Motivated by behavioral economics, CPT combines an asymmetric utility transformation around a reference point with probability distortion. Building on our theorem, we design a first-order policy gradient algorithm for CPT-RL using a Monte Carlo gradient estimator based on order statistics. We establish statistical guarantees for the estimator and prove asymptotic convergence of the resulting algorithm to first-order stationary points of the (generally non-convex) CPT objective. Simulations illustrate qualitative behaviors induced by CPT and compare our first-order approach to existing zeroth-order methods.