A framework for causal segmentation analysis with machine learning in large-scale digital experiments

TL;DR

This paper introduces a comprehensive, model-agnostic framework for identifying user segments with differential treatment effects in large-scale digital experiments, enabling targeted treatment assignment and causal impact evaluation.

Contribution

It presents a novel, unified methodology combining causal inference and machine learning for causal segmentation analysis, with an open-source R package implementation.

Findings

Effective identification of beneficial user segments

Capability to evaluate dynamic treatment assignment impacts

Applicable to both randomized and quasi-experiments

Abstract

We present an end-to-end methodological framework for causal segment discovery that aims to uncover differential impacts of treatments across subgroups of users in large-scale digital experiments. Building on recent developments in causal inference and non/semi-parametric statistics, our approach unifies two objectives: (1) the discovery of user segments that stand to benefit from a candidate treatment based on subgroup-specific treatment effects, and (2) the evaluation of causal impacts of dynamically assigning units to a study's treatment arm based on their predicted segment-specific benefit or harm. Our proposal is model-agnostic, capable of incorporating state-of-the-art machine learning algorithms into the estimation procedure, and is applicable in randomized A/B tests and quasi-experiments. An open source R package implementation, sherlock, is introduced.