Inverse Bayesian Optimization: Learning Human Acquisition Functions in an Exploration vs Exploitation Search Task

TL;DR

This paper presents a probabilistic framework to infer human acquisition functions in exploration-exploitation tasks by modeling observed behavior as samples from Bayesian optimization, improving understanding of human decision-making.

Contribution

It introduces a novel inverse Bayesian optimization method to estimate human acquisition functions from behavioral data, allowing for deviations and augmentations to standard models.

Findings

Many subjects show exploration preferences beyond standard acquisition functions.

Augmented acquisition functions better fit human behavior in the task.

The framework enables inference of individual human strategies in optimization tasks.

Abstract

This paper introduces a probabilistic framework to estimate parameters of an acquisition function given observed human behavior that can be modeled as a collection of sample paths from a Bayesian optimization procedure. The methodology involves defining a likelihood on observed human behavior from an optimization task, where the likelihood is parameterized by a Bayesian optimization subroutine governed by an unknown acquisition function. This structure enables us to make inference on a subject's acquisition function while allowing their behavior to deviate around the solution to the Bayesian optimization subroutine. To test our methods, we designed a sequential optimization task which forced subjects to balance exploration and exploitation in search of an invisible target location. Applying our proposed methods to the resulting data, we find that many subjects tend to exhibit exploration preferences beyond that of standard acquisition functions to capture. Guided by the model discrepancies, we augment the candidate acquisition functions to yield a superior fit to the human behavior in this task.