The Transfer Performance of Economic Models

TL;DR

This paper introduces a framework for evaluating how well economic models predict in new domains, comparing their transferability to black-box algorithms through finite-sample forecast intervals.

Contribution

It provides a tractable formulation for out-of-domain prediction and derives finite-sample forecast intervals to assess transfer errors of economic models.

Findings

Black-box algorithms outperform economic models within the same domain.

Economic models generalize better across different domains than black-box algorithms.

Finite-sample forecast intervals reliably cover transfer errors with specified probability.

Abstract

Economists often estimate models using data from a particular domain, e.g. estimating risk preferences in a particular subject pool or for a specific class of lotteries. Whether a model's predictions extrapolate well across domains depends on whether the estimated model has captured generalizable structure. We provide a tractable formulation for this "out-of-domain" prediction problem and define the transfer error of a model based on how well it performs on data from a new domain. We derive finite-sample forecast intervals that are guaranteed to cover realized transfer errors with a user-selected probability when domains are iid, and use these intervals to compare the transferability of economic models and black box algorithms for predicting certainty equivalents. We find that in this application, the black box algorithms we consider outperform standard economic models when estimated and tested on data from the same domain, but the economic models generalize across domains better than the black-box algorithms do.