Predicting human decisions with behavioral theories and machine learning

TL;DR

This paper introduces BEAST-GB, a hybrid machine learning model that combines behavioral theory with gradient boosting to improve the prediction of human decisions under risk, outperforming existing models and generalizing well across contexts.

Contribution

The paper presents BEAST-GB, a novel hybrid model integrating behavioral theory with machine learning, demonstrating superior predictive accuracy and robustness over existing models.

Findings

BEAST-GB won the CPC18 competition for predicting risky choices.

BEAST-GB outperforms neural networks and behavioral models in accuracy.

BEAST-GB generalizes effectively across unseen experimental contexts.

Abstract

Predicting human decisions under risk and uncertainty remains a fundamental challenge across disciplines. Existing models often struggle even in highly stylized tasks like choice between lotteries. We introduce BEAST Gradient Boosting (BEAST-GB), a hybrid model integrating behavioral theory (BEAST) with machine learning. We first present CPC18, a competition for predicting risky choice, in which BEAST-GB won. Then, using two large datasets, we demonstrate BEAST-GB predicts more accurately than neural networks trained on extensive data and dozens of existing behavioral models. BEAST-GB also generalizes robustly across unseen experimental contexts, surpassing direct empirical generalization, and helps refine and improve the behavioral theory itself. Our analyses highlight the potential of anchoring predictions on behavioral theory even in data-rich settings and even when the theory alone falters. Our results underscore how integrating machine learning with theoretical frameworks, especially those-like BEAST-designed for prediction, can improve our ability to predict and understand human behavior.