Robust Contextual Bandit via the Capped-$\ell_{2}$ norm

TL;DR

This paper introduces a robust actor-critic contextual bandit method using the capped-$\\ell_{2}$ norm to effectively handle outliers in mHealth interventions, improving decision-making robustness in noisy environments.

Contribution

The paper proposes a novel robust actor-critic algorithm with capped-$\ell_{2}$ norm for outlier resistance, including a method to set its key parameter based on statistical outlier definitions.

Findings

Achieves similar performance to state-of-the-art methods on clean data.

Significantly outperforms existing methods on outlier-contaminated data.

Enhances robustness of mHealth decision-making processes.

Abstract

This paper considers the actor-critic contextual bandit for the mobile health (mHealth) intervention. The state-of-the-art decision-making methods in mHealth generally assume that the noise in the dynamic system follows the Gaussian distribution. Those methods use the least-square-based algorithm to estimate the expected reward, which is prone to the existence of outliers. To deal with the issue of outliers, we propose a novel robust actor-critic contextual bandit method for the mHealth intervention. In the critic updating, the capped-$\ell_{2}$ norm is used to measure the approximation error, which prevents outliers from dominating our objective. A set of weights could be achieved from the critic updating. Considering them gives a weighted objective for the actor updating. It provides the badly noised sample in the critic updating with zero weights for the actor updating. As a result, the robustness of both actor-critic updating is enhanced. There is a key parameter in the capped-$\ell_{2}$ norm. We provide a reliable method to properly set it by making use of one of the most fundamental definitions of outliers in statistics. Extensive experiment results demonstrate that our method can achieve almost identical results compared with the state-of-the-art methods on the dataset without outliers and dramatically outperform them on the datasets noised by outliers.