Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information

TL;DR

This paper develops an offline reinforcement learning framework for human-guided human-machine interactions involving private information, addressing confounding bias and distribution mismatch to optimize policies in a two-player game setting.

Contribution

It introduces a novel instrumental variable approach to handle confounding bias and proposes an off-policy evaluation and learning algorithm leveraging pessimism for two-player offline RL with private info.

Findings

The proposed method effectively adjusts for unmeasured confounding bias.

The off-policy learning algorithm converges to the optimal policy pair under mild conditions.

The approach demonstrates promising potential for privacy-preserving human-machine interaction optimization.

Abstract

Motivated by the human-machine interaction such as training chatbots for improving customer satisfaction, we study human-guided human-machine interaction involving private information. We model this interaction as a two-player turn-based game, where one player (Alice, a human) guides the other player (Bob, a machine) towards a common goal. Specifically, we focus on offline reinforcement learning (RL) in this game, where the goal is to find a policy pair for Alice and Bob that maximizes their expected total rewards based on an offline dataset collected a priori. The offline setting presents two challenges: (i) We cannot collect Bob's private information, leading to a confounding bias when using standard RL methods, and (ii) a distributional mismatch between the behavior policy used to collect data and the desired policy we aim to learn. To tackle the confounding bias, we treat Bob's previous action as an instrumental variable for Alice's current decision making so as to adjust for the unmeasured confounding. We develop a novel identification result and use it to propose a new off-policy evaluation (OPE) method for evaluating policy pairs in this two-player turn-based game. To tackle the distributional mismatch, we leverage the idea of pessimism and use our OPE method to develop an off-policy learning algorithm for finding a desirable policy pair for both Alice and Bob. Finally, we prove that under mild assumptions such as partial coverage of the offline data, the policy pair obtained through our method converges to the optimal one at a satisfactory rate.