Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL

TL;DR

This paper introduces a unified, flexible framework for modeling human decision-making across bandits and reinforcement learning, inspired by clinical insights into neurological and psychiatric disorders, demonstrating its effectiveness on various simulated and real-world tasks.

Contribution

It proposes a novel two-stream reward processing model that unifies multiple decision-making paradigms and incorporates clinical insights to better mimic human behaviors.

Findings

The framework can model behaviors in MAB, CB, and RL settings.

Clinically-inspired agents show realistic behavioral trajectories.

The model performs well on simulated, real-world, and game-based tasks.

Abstract

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting.