Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies

TL;DR

This paper introduces a novel inverse online learning framework that interprets and analyzes non-stationary decision policies over time, providing insights into how agents adapt their actions based on perceived effects.

Contribution

It presents a new approach to inverse online learning using deep state-space models to interpret non-stationary, reactionary policies in decision-making processes.

Findings

Effective retrospective estimation of perceived effects.

Insights into decision process dynamics over time.

Application to UNOS organ donation decisions.

Abstract

Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decision-maker's ability to perform a task, e.g. to alert them to potential biases or oversights on their part. To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience. To then understand the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.