Effective Monitoring of Online Decision-Making Algorithms in Digital Intervention Implementation
Anna L. Trella, Susobhan Ghosh, Erin E. Bonar, Lara Coughlin, Finale, Doshi-Velez, Yongyi Guo, Pei-Yao Hung, Inbal Nahum-Shani, Vivek Shetty,, Maureen Walton, Iris Yan, Kelly W. Zhang, Susan A. Murphy

TL;DR
This paper offers practical guidelines and case studies for monitoring online decision-making algorithms in digital health interventions to ensure safety, data quality, and effective treatment delivery.
Contribution
It introduces specific monitoring strategies, including fallback procedures and issue severity categorization, demonstrated through two clinical trial case studies.
Findings
Monitoring detected real-time issues like memory and communication failures.
Fallback methods prevented treatment gaps and data errors.
Guidelines increased confidence in deploying online algorithms in interventions.
Abstract
Online AI decision-making algorithms are increasingly used by digital interventions to dynamically personalize treatment to individuals. These algorithms determine, in real-time, the delivery of treatment based on accruing data. The objective of this paper is to provide guidelines for enabling effective monitoring of online decision-making algorithms with the goal of (1) safeguarding individuals and (2) ensuring data quality. We elucidate guidelines and discuss our experience in monitoring online decision-making algorithms in two digital intervention clinical trials (Oralytics and MiWaves). Our guidelines include (1) developing fallback methods, pre-specified procedures executed when an issue occurs, and (2) identifying potential issues categorizing them by severity (red, yellow, and green). Across both trials, the monitoring systems detected real-time issues such as out-of-memory…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEducational Innovations and Challenges
