Evaluating Conditional Cash Transfer Policies with Machine Learning Methods

TL;DR

This study compares machine learning models and a structural econometric model in predicting the effects of cash transfer policies, finding machine learning excels in out-of-sample predictions with ample data, but structural models perform better with limited data.

Contribution

It provides an empirical comparison of machine learning and structural models in economic policy prediction, highlighting conditions where each approach is preferable.

Findings

Machine learning models outperform the structural model in out-of-sample forecasts.

Random forest and adaboost achieve the highest accuracy among machine learning methods.

Structural models perform better in long-term within-sample simulations.

Abstract

This paper presents an out-of-sample prediction comparison between major machine learning models and the structural econometric model. Over the past decade, machine learning has established itself as a powerful tool in many prediction applications, but this approach is still not widely adopted in empirical economic studies. To evaluate the benefits of this approach, I use the most common machine learning algorithms, CART, C4.5, LASSO, random forest, and adaboost, to construct prediction models for a cash transfer experiment conducted by the Progresa program in Mexico, and I compare the prediction results with those of a previous structural econometric study. Two prediction tasks are performed in this paper: the out-of-sample forecast and the long-term within-sample simulation. For the out-of-sample forecast, both the mean absolute error and the root mean square error of the school attendance rates found by all machine learning models are smaller than those found by the structural model. Random forest and adaboost have the highest accuracy for the individual outcomes of all subgroups. For the long-term within-sample simulation, the structural model has better performance than do all of the machine learning models. The poor within-sample fitness of the machine learning model results from the inaccuracy of the income and pregnancy prediction models. The result shows that the machine learning model performs better than does the structural model when there are many data to learn; however, when the data are limited, the structural model offers a more sensible prediction. The findings of this paper show promise for adopting machine learning in economic policy analyses in the era of big data.