Non-Plug-In Estimators Could Outperform Plug-In Estimators: a Cautionary Note and a Diagnosis

TL;DR

This paper challenges the common belief that plug-in estimators like TMLE outperform non-plug-in estimators such as DML in small samples, showing through simulations that DML can perform better and suggesting diagnostic checks for TMLE stability.

Contribution

It provides a simulation-based comparison of TMLE and DML estimators, highlighting scenarios where TMLE may underperform and proposing fluctuation magnitude as a diagnostic tool.

Findings

DML outperforms some TMLE versions in small samples

TMLE fluctuations are unstable and can indicate poor performance

Checking fluctuation magnitude can diagnose TMLE reliability

Abstract

Objectives: Highly flexible nonparametric estimators have gained popularity in causal inference and epidemiology. Popular examples of such estimators include targeted maximum likelihood estimators (TMLE) and double machine learning (DML). TMLE is often argued or suggested to be better than DML estimators and several other estimators in small to moderate samples -- even if they share the same large-sample properties -- because TMLE is a plug-in estimator and respects the known bounds on the parameter, while other estimators might fall outside the known bounds and yield absurd estimates. However, this argument is not a rigorously proven result and may fail in certain cases. Methods: In a carefully chosen simulation setting, I compare the performance of several versions of TMLE and DML estimators of the average treatment effect among treated in small to moderate samples. Results: In this simulation setting, DML estimators outperforms some versions of TMLE in small samples. TMLE fluctuations are unstable, and hence empirically checking the magnitude of the TMLE fluctuation might alert cases where TMLE might perform poorly. Conclusions: As a plug-in estimator, TMLE is not guaranteed to outperform non-plug-in counterparts such as DML estimators in small samples. Checking the fluctuation magnitude might be a useful diagnosis for TMLE. More rigorous theoretical justification is needed to understand and compare the finite-sample performance of these highly flexible estimators in general.