Reevaluating Causal Estimation Methods with Data from a Product Release

TL;DR

This paper evaluates the effectiveness of modern causal estimation methods using real-world data from a product feature rollout, emphasizing the importance of careful modeling for accurate causal effect recovery.

Contribution

It provides empirical analysis of causal methods on actual product data, offering best practices for credible treatment effect estimation in high-dimensional settings.

Findings

Ground truth causal effects can be recovered with careful modeling.

Modern causal methods are feasible but require specific modeling choices.

The study builds on LaLonde's observational causal literature to improve credibility.

Abstract

Recent developments in causal machine learning methods have made it easier to estimate flexible relationships between confounders, treatments and outcomes, making unconfoundedness assumptions in causal analysis more palatable. How successful are these approaches in recovering ground truth baselines? In this paper we analyze a new data sample including an experimental rollout of a new feature at a large technology company and a simultaneous sample of users who endogenously opted into the feature. We find that recovering ground truth causal effects is feasible -- but only with careful modeling choices. Our results build on the observational causal literature beginning with LaLonde (1986), offering best practices for more credible treatment effect estimation in modern, high-dimensional datasets.