Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Management

TL;DR

This paper introduces a continuous-time reinforcement learning framework for intensity control in network revenue management, leveraging event-driven structures to avoid time discretization and demonstrating superior performance over existing methods.

Contribution

The study develops a novel continuous-time RL approach for intensity control, specifically tailored for choice-based network revenue management, with algorithms adapted for event-driven dynamics.

Findings

The proposed method outperforms state-of-the-art benchmarks.

It scales effectively to large, complex problems.

It maintains computational efficiency comparable to discretization-based methods.

Abstract

Intensity control is a class of continuous-time dynamic optimization problems with many important applications in Operations Research including queueing and revenue management. In this study, we propose a practical continuous-time reinforcement learning framework for intensity control using choice-based network revenue management as a case study, which is a classical problem in revenue management that features a large state space, a large action space, and a continuous time horizon. We show that by leveraging the event-driven structure of the problem and the inherent discretization of sample paths created by the state-jump times, a defining feature of intensity control, one does not need to discretize the time horizon in advance. We adapt discrete-time Monte Carlo and temporal difference learning algorithms for policy evaluation to continuous time and develop policy-gradient-based actor-critic algorithms for event-driven intensity control. Through a comprehensive numerical study, we evaluate the proposed approach against various state-of-the-art benchmarks, demonstrating its overall superior performance and effective scalability to large-scale problems. Notably, compared to discretization-based reinforcement learning methods, our continuous-time approach delivers significantly superior performance while maintaining comparable computational efficiency. This advantage is particularly pronounced in highly non-stationary environments.