Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization

TL;DR

This paper introduces a Bayesian extension to the Act-Then-Measure heuristic for reinforcement learning in mobile health, improving stability and efficiency in low-data, noisy environments, but highlighting challenges in complex real-world settings.

Contribution

It proposes a Bayesian ATM algorithm using Kalman filter-style updates, enhancing stability and sample efficiency over standard Q-learning in certain environments.

Findings

Bayesian ATM reduces variance and improves stability in small environments.

Both ATM variants perform poorly in complex, real-world mHealth settings.

Uncertainty-aware methods are valuable in low-data, noisy environments but need adaptation for complex domains.

Abstract

Reinforcement learning in mobile health (mHealth) interventions requires balancing intervention efficacy with user burden, particularly when state measurements (for example, user surveys or feedback) are costly yet essential. The Act-Then-Measure (ATM) heuristic addresses this challenge by decoupling control and measurement actions within the Action-Contingent Noiselessly Observable Markov Decision Process (ACNO-MDP) framework. However, the standard ATM algorithm relies on a temporal-difference-inspired Q-learning method, which is prone to instability in sparse and noisy environments. In this work, we propose a Bayesian extension to ATM that replaces standard Q-learning with a Kalman filter-style Bayesian update, maintaining uncertainty-aware estimates of Q-values and enabling more stable and sample-efficient learning. We evaluate our method in both toy environments and clinically motivated testbeds. In small, tabular environments, Bayesian ATM achieves comparable or improved scalarized returns with substantially lower variance and more stable policy behavior. In contrast, in larger and more complex mHealth settings, both the standard and Bayesian ATM variants perform poorly, suggesting a mismatch between ATM's modeling assumptions and the structural challenges of real-world mHealth domains. These findings highlight the value of uncertainty-aware methods in low-data settings while underscoring the need for new RL algorithms that explicitly model causal structure, continuous states, and delayed feedback under observation cost constraints.