The Missing Link: Allocation Performance in Causal Machine Learning

TL;DR

This paper investigates how challenges like distribution shifts affect the performance of causal ML models in real-world decision-making, emphasizing the variability across different scenarios in social environments.

Contribution

It provides empirical insights into the impact of distribution shifts on CATE model performance in complex social decision-making contexts.

Findings

Performance of CATE models varies significantly across scenarios

Distribution shifts notably influence prediction accuracy

Allocation effectiveness is sensitive to environmental challenges

Abstract

Automated decision-making (ADM) systems are being deployed across a diverse range of critical problem areas such as social welfare and healthcare. Recent work highlights the importance of causal ML models in ADM systems, but implementing them in complex social environments poses significant challenges. Research on how these challenges impact the performance in specific downstream decision-making tasks is limited. Addressing this gap, we make use of a comprehensive real-world dataset of jobseekers to illustrate how the performance of a single CATE model can vary significantly across different decision-making scenarios and highlight the differential influence of challenges such as distribution shifts on predictions and allocations.