Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environments

TL;DR

This paper introduces super reinforcement learning, leveraging human-AI interaction to improve decision-making policies in confounded environments, guaranteeing performance enhancements over traditional methods.

Contribution

It proposes a novel super reinforcement learning paradigm that incorporates observed actions to outperform standard policies, with theoretical guarantees and practical algorithms.

Findings

Super policies outperform standard optimal policies.

Inclusion of past actions provides valuable insights in confounded settings.

Algorithms demonstrate strong theoretical and empirical performance.

Abstract

As AI becomes more prevalent throughout society, effective methods of integrating humans and AI systems that leverage their respective strengths and mitigate risk have become an important priority. In this paper, we introduce the paradigm of super reinforcement learning that takes advantage of Human-AI interaction for data driven sequential decision making. This approach utilizes the observed action, either from AI or humans, as input for achieving a stronger oracle in policy learning for the decision maker (humans or AI). In the decision process with unmeasured confounding, the actions taken by past agents can offer valuable insights into undisclosed information. By including this information for the policy search in a novel and legitimate manner, the proposed super reinforcement learning will yield a super-policy that is guaranteed to outperform both the standard optimal policy and the behavior one (e.g., past agents' actions). We call this stronger oracle a blessing from human-AI interaction. Furthermore, to address the issue of unmeasured confounding in finding super-policies using the batch data, a number of nonparametric and causal identifications are established. Building upon on these novel identification results, we develop several super-policy learning algorithms and systematically study their theoretical properties such as finite-sample regret guarantee. Finally, we illustrate the effectiveness of our proposal through extensive simulations and real-world applications.