Pay No Attention to the Model Behind the Curtain

TL;DR

The paper critically examines the reliability and real-world relevance of widely used models, highlighting their often superficial connection to actual phenomena and potential conflicts with reality in policy contexts.

Contribution

It exposes the limitations of common modeling practices and emphasizes the importance of scrutinizing model assumptions and their alignment with real-world data.

Findings

Models often produce numbers without real-world grounding

Scaling and uncertainty assumptions can conflict with reality

Many model parameters do not measure what they claim to

Abstract

Many widely used models amount to an elaborate means of making up numbers--but once a number has been produced, it tends to be taken seriously and its source (the model) is rarely examined carefully. Many widely used models have little connection to the real-world phenomena they purport to explain. Common steps in modeling to support policy decisions, such as putting disparate things on the same scale, may conflict with reality. Not all costs and benefits can be put on the same scale, not all uncertainties can be expressed as probabilities, and not all model parameters measure what they purport to measure. These ideas are illustrated with examples from seismology, wind-turbine bird deaths, soccer penalty cards, gender bias in academia, and climate policy.