Invariant Policy Learning: A Causal Perspective

TL;DR

This paper introduces a causal perspective on invariant policy learning in contextual bandits, addressing environmental shifts by proposing methods that ensure policy robustness across changing mechanisms.

Contribution

It develops a framework integrating causality and invariance into offline contextual bandits to handle environment changes, introducing the concept of policy invariance.

Findings

Optimal invariant policies can generalize across environments under certain assumptions.

Establishes connections between causality, invariance, and contextual bandits.

Addresses the challenge of environmental shifts in high-stakes applications.

Abstract

Contextual bandit and reinforcement learning algorithms have been successfully used in various interactive learning systems such as online advertising, recommender systems, and dynamic pricing. However, they have yet to be widely adopted in high-stakes application domains, such as healthcare. One reason may be that existing approaches assume that the underlying mechanisms are static in the sense that they do not change over different environments. In many real-world systems, however, the mechanisms are subject to shifts across environments which may invalidate the static environment assumption. In this paper, we take a step toward tackling the problem of environmental shifts considering the framework of offline contextual bandits. We view the environmental shift problem through the lens of causality and propose multi-environment contextual bandits that allow for changes in the underlying mechanisms. We adopt the concept of invariance from the causality literature and introduce the notion of policy invariance. We argue that policy invariance is only relevant if unobserved variables are present and show that, in that case, an optimal invariant policy is guaranteed to generalize across environments under suitable assumptions. Our results establish concrete connections among causality, invariance, and contextual bandits.