Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care

TL;DR

This paper presents an online reinforcement learning algorithm designed to optimize personalized prompts in a mobile app to improve oral hygiene behaviors, addressing delayed effects and user burden in real-world settings.

Contribution

It introduces a reward design that balances health outcomes and user burden, and a hyperparameter optimization procedure using a simulation test bed.

Findings

Effective reward function for oral health improvement

Hyperparameter tuning via simulation enhances algorithm performance

Algorithm deployed in a real-world oral self-care app

Abstract

Dental disease is one of the most common chronic diseases despite being largely preventable. However, professional advice on optimal oral hygiene practices is often forgotten or abandoned by patients. Therefore patients may benefit from timely and personalized encouragement to engage in oral self-care behaviors. In this paper, we develop an online reinforcement learning (RL) algorithm for use in optimizing the delivery of mobile-based prompts to encourage oral hygiene behaviors. One of the main challenges in developing such an algorithm is ensuring that the algorithm considers the impact of the current action on the effectiveness of future actions (i.e., delayed effects), especially when the algorithm has been made simple in order to run stably and autonomously in a constrained, real-world setting (i.e., highly noisy, sparse data). We address this challenge by designing a quality reward which maximizes the desired health outcome (i.e., high-quality brushing) while minimizing user burden. We also highlight a procedure for optimizing the hyperparameters of the reward by building a simulation environment test bed and evaluating candidates using the test bed. The RL algorithm discussed in this paper will be deployed in Oralytics, an oral self-care app that provides behavioral strategies to boost patient engagement in oral hygiene practices.