Reinforcement Learning Under Algorithmic Triage

TL;DR

This paper introduces a reinforcement learning framework for algorithmic triage, combining offline and on-policy training to optimize machine-human collaboration, demonstrated through synthetic driving simulations.

Contribution

It develops a novel two-stage actor-critic method for reinforcement learning under triage, integrating offline human data with on-policy adjustments.

Findings

The two-stage method improves collaboration between machine and human policies.

Models trained with this approach outperform several baselines in synthetic driving tasks.

The approach effectively adapts to the impact of switching between human and machine decisions.

Abstract

Methods to learn under algorithmic triage have predominantly focused on supervised learning settings where each decision, or prediction, is independent of each other. Under algorithmic triage, a supervised learning model predicts a fraction of the instances and humans predict the remaining ones. In this work, we take a first step towards developing reinforcement learning models that are optimized to operate under algorithmic triage. To this end, we look at the problem through the framework of options and develop a two-stage actor-critic method to learn reinforcement learning models under triage. The first stage performs offline, off-policy training using human data gathered in an environment where the human has operated on their own. The second stage performs on-policy training to account for the impact that switching may have on the human policy, which may be difficult to anticipate from the above human data. Extensive simulation experiments in a synthetic car driving task show that the machine models and the triage policies trained using our two-stage method effectively complement human policies and outperform those provided by several competitive baselines.