Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study

TL;DR

This study empirically evaluates how hyperparameter tuning and ML method choices affect causal inference accuracy using Double Machine Learning, providing practical guidance for optimizing causal estimates in simulation settings.

Contribution

It offers new empirical insights into hyperparameter tuning's impact on causal estimation with DML and compares different ML methods and AutoML frameworks.

Findings

Data splitting schemes significantly influence causal estimation accuracy.

Choice of ML methods and hyperparameters affects the quality of causal estimates.

Predictive performance metrics can inform causal model selection.

Abstract

Proper hyperparameter tuning is essential for achieving optimal performance of modern machine learning (ML) methods in predictive tasks. While there is an extensive literature on tuning ML learners for prediction, there is only little guidance available on tuning ML learners for causal machine learning and how to select among different ML learners. In this paper, we empirically assess the relationship between the predictive performance of ML methods and the resulting causal estimation based on the Double Machine Learning (DML) approach by Chernozhukov et al. (2018). DML relies on estimating so-called nuisance parameters by treating them as supervised learning problems and using them as plug-in estimates to solve for the (causal) parameter. We conduct an extensive simulation study using data from the 2019 Atlantic Causal Inference Conference Data Challenge. We provide empirical insights on the role of hyperparameter tuning and other practical decisions for causal estimation with DML. First, we assess the importance of data splitting schemes for tuning ML learners within Double Machine Learning. Second, we investigate how the choice of ML methods and hyperparameters, including recent AutoML frameworks, impacts the estimation performance for a causal parameter of interest. Third, we assess to what extent the choice of a particular causal model, as characterized by incorporated parametric assumptions, can be based on predictive performance metrics.