Robust Tests in Online Decision-Making

TL;DR

This paper introduces a robust actor-critic bandit algorithm and a new testing procedure to evaluate actor parameters, addressing critic misspecification issues in sequential decision-making, especially relevant for mobile health applications.

Contribution

It proposes a modified actor-critic algorithm that is robust to critic misspecification and develops a novel testing procedure for actor parameters under such conditions.

Findings

The new algorithm maintains performance despite critic misspecification.

The testing procedure provides valid inference even with model misspecification.

Application to mobile health demonstrates practical utility.

Abstract

Bandit algorithms are widely used in sequential decision problems to maximize the cumulative reward. One potential application is mobile health, where the goal is to promote the user's health through personalized interventions based on user specific information acquired through wearable devices. Important considerations include the type of, and frequency with which data is collected (e.g. GPS, or continuous monitoring), as such factors can severely impact app performance and users' adherence. In order to balance the need to collect data that is useful with the constraint of impacting app performance, one needs to be able to assess the usefulness of variables. Bandit feedback data are sequentially correlated, so traditional testing procedures developed for independent data cannot apply. Recently, a statistical testing procedure was developed for the actor-critic bandit algorithm. An actor-critic algorithm maintains two separate models, one for the actor, the action selection policy, and the other for the critic, the reward model. The performance of the algorithm as well as the validity of the test are guaranteed only when the critic model is correctly specified. However, misspecification is frequent in practice due to incorrect functional form or missing covariates. In this work, we propose a modified actor-critic algorithm which is robust to critic misspecification and derive a novel testing procedure for the actor parameters in this case.