Practical considerations when designing an online learning algorithm for an app-based mHealth intervention

TL;DR

This paper discusses practical challenges and solutions in designing reinforcement learning algorithms for app-based mHealth interventions, illustrated through the LowSalt4Life 2 trial targeting sodium reduction.

Contribution

It provides a template of solutions for key challenges in deploying reinforcement learning in mHealth trials, including reward definition, timescale, modeling, and data handling.

Findings

Developed a reinforcement learning algorithm for app engagement.

Identified key challenges and proposed template solutions.

Enhanced understanding of deploying RL in real-world mHealth trials.

Abstract

The ubiquitous nature of mobile health (mHealth) technology has expanded opportunities for the integration of reinforcement learning into traditional clinical trial designs, allowing researchers to learn individualized treatment policies during the study. LowSalt4Life 2 (LS4L2) is a recent trial aimed at reducing sodium intake among hypertensive individuals through an app-based intervention. A reinforcement learning algorithm, which was deployed in one of the trial arms, was designed to send reminder notifications to promote app engagement in contexts where the notification would be effective, i.e., when a participant is likely to open the app in the next 30-minute and not when prior data suggested reduced effectiveness. Such an algorithm can improve app-based mHealth interventions by reducing participant burden and more effectively promoting behavior change. We encountered various challenges during the implementation of the learning algorithm, which we present as a template to solving challenges in future trials that deploy reinforcement learning algorithms. We provide template solutions based on LS4L2 for solving the key challenges of (i) defining a relevant reward, (ii) determining a meaningful timescale for optimization, (iii) specifying a robust statistical model that allows for automation, (iv) balancing model flexibility with computational cost, and (v) addressing missing values in gradually collected data.