Scalable Decisions using a Bayesian Decision-Theoretic Approach

TL;DR

This paper introduces a Bayesian decision-theoretic framework for scalable, multi-objective decision-making in policy experiments, improving efficiency and consistency over traditional methods by integrating prior knowledge and business trade-offs.

Contribution

It presents a novel hierarchical Bayesian approach that systematically incorporates multiple objectives and historical data to enhance decision-making in policy experiments.

Findings

Increased estimation efficiency with hierarchical priors.

Simplified decision process by integrating business preferences.

Validated effectiveness through Amazon supply chain experiments.

Abstract

Randomized controlled experiments assess new policy impacts on performance metrics to inform launch decisions. Traditional approaches evaluate metrics independently despite correlations, and mixed results (e.g., positive revenue impact, negative customer experience) require manual judgment, hindering scalability. We propose a Bayesian decision-theoretic framework that systematically incorporates multiple objectives and trade-offs by comparing expected risks across decisions. Our approach combines experimenter-defined loss functions with observed evidence, using hierarchical models to leverage historical experiment learnings for prior information on treatment effects. Through real and simulated Amazon supply chain experiments, we demonstrate that compared to null hypothesis statistical testing, our method increases estimation efficiency via informative hierarchical priors and simplifies decision-making by systematically incorporating business preferences and costs for comprehensive, scalable decisions.