Optimal design of experiments to identify latent behavioral types

TL;DR

This paper introduces computational improvements to Bayesian optimal experiment design, enabling efficient identification of behavioral models, demonstrated through a game experiment showing optimal designs outperform expert-chosen ones in model discrimination.

Contribution

The paper extends a Bayesian optimal experiment design method with two computational enhancements, making it more tractable for practical applications.

Findings

Optimal design reduces data needed to distinguish models

Reinforcement learning best explains human decisions

Behavior not fully captured by Bayesian Nash equilibrium

Abstract

Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a "wisdom of experts" prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.