A Comparative Tutorial of Bayesian Sequential Design and Reinforcement Learning

TL;DR

This paper compares Bayesian sequential design and reinforcement learning, highlighting their similarities, differences, and applications in adaptive clinical trial design, with practical examples illustrating their use in sequential stopping problems.

Contribution

It provides a comparative analysis of RL and Bayesian sequential design, demonstrating their connections and discussing their respective strengths and limitations in clinical trial contexts.

Findings

RL and BSD share many similarities in sequential decision problems.

Both approaches can be applied to adaptive clinical trial design.

Practical examples illustrate the implementation and results of each method.

Abstract

Reinforcement Learning (RL) is a computational approach to reward-driven learning in sequential decision problems. It implements the discovery of optimal actions by learning from an agent interacting with an environment rather than from supervised data. We contrast and compare RL with traditional sequential design, focusing on simulation-based Bayesian sequential design (BSD). Recently, there has been an increasing interest in RL techniques for healthcare applications. We introduce two related applications as motivating examples. In both applications, the sequential nature of the decisions is restricted to sequential stopping. Rather than a comprehensive survey, the focus of the discussion is on solutions using standard tools for these two relatively simple sequential stopping problems. Both problems are inspired by adaptive clinical trial design. We use examples to explain the terminology and mathematical background that underlie each framework and map one to the other. The implementations and results illustrate the many similarities between RL and BSD. The results motivate the discussion of the potential strengths and limitations of each approach.