Non-stationary Delayed Combinatorial Semi-Bandit with Causally Related Rewards

TL;DR

This paper addresses the challenge of decision-making in non-stationary, delayed, combinatorial semi-bandit environments with causally related rewards, proposing a new algorithm that learns causal structures to improve long-term reward maximization.

Contribution

It introduces a novel policy that learns causal dependencies from delayed feedback in non-stationary settings, with proven regret bounds and empirical validation.

Findings

The proposed method effectively learns causal relations despite delays.

The algorithm adapts to environmental drifts while optimizing rewards.

Numerical analysis demonstrates practical utility in real-world data.

Abstract

Sequential decision-making under uncertainty is often associated with long feedback delays. Such delays degrade the performance of the learning agent in identifying a subset of arms with the optimal collective reward in the long run. This problem becomes significantly challenging in a non-stationary environment with structural dependencies amongst the reward distributions associated with the arms. Therefore, besides adapting to delays and environmental changes, learning the causal relations alleviates the adverse effects of feedback delay on the decision-making process. We formalize the described setting as a non-stationary and delayed combinatorial semi-bandit problem with causally related rewards. We model the causal relations by a directed graph in a stationary structural equation model. The agent maximizes the long-term average payoff, defined as a linear function of the base arms' rewards. We develop a policy that learns the structural dependencies from delayed feedback and utilizes that to optimize the decision-making while adapting to drifts. We prove a regret bound for the performance of the proposed algorithm. Besides, we evaluate our method via numerical analysis using synthetic and real-world datasets to detect the regions that contribute the most to the spread of Covid-19 in Italy.