Harnessing Causality in Reinforcement Learning With Bagged Decision Times

TL;DR

This paper introduces an online reinforcement learning method that leverages causal DAGs to handle non-Markovian, non-stationary decision problems with bagged decision times, demonstrated on mobile health data.

Contribution

It develops a novel RL algorithm using causal DAGs to construct Markov states in non-Markovian, non-stationary settings with bagged decision times.

Findings

The proposed method effectively maximizes rewards in mobile health data.

States constructed achieve maximal optimal value among all periodic MDP states.

The approach handles non-Markovian and non-stationary dynamics successfully.

Abstract

We consider reinforcement learning (RL) for a class of problems with bagged decision times. A bag contains a finite sequence of consecutive decision times. The transition dynamics are non-Markovian and non-stationary within a bag. All actions within a bag jointly impact a single reward, observed at the end of the bag. For example, in mobile health, multiple activity suggestions in a day collectively affect a user's daily commitment to being active. Our goal is to develop an online RL algorithm to maximize the discounted sum of the bag-specific rewards. To handle non-Markovian transitions within a bag, we utilize an expert-provided causal directed acyclic graph (DAG). Based on the DAG, we construct states as a dynamical Bayesian sufficient statistic of the observed history, which results in Markov state transitions within and across bags. We then formulate this problem as a periodic Markov decision process (MDP) that allows non-stationarity within a period. An online RL algorithm based on Bellman equations for stationary MDPs is generalized to handle periodic MDPs. We show that our constructed state achieves the maximal optimal value function among all state constructions for a periodic MDP. Finally, we evaluate the proposed method on testbed variants built from real data in a mobile health clinical trial.