The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisation

TL;DR

This paper provides a formal analysis of Goodhart's law, examining how dependence between proxy metrics and goals affects optimization outcomes, especially under different tail distribution assumptions, and introduces a paradigm-agnostic framework.

Contribution

It relaxes previous independence assumptions and offers a formal, general framework to understand Goodhart's law across various learning paradigms and dependence scenarios.

Findings

Dependence does not alter Goodhart's effect with light-tailed goal and discrepancy.

Heavy-tailed discrepancy can cause over-optimization inversely proportional to tail heaviness.

The framework is paradigm-agnostic and applicable to diverse settings.

Abstract

Goodhart's law is a famous adage in policy-making that states that ``When a measure becomes a target, it ceases to be a good measure''. As machine learning models and the optimisation capacity to train them grow, growing empirical evidence reinforced the belief in the validity of this law without however being formalised. Recently, a few attempts were made to formalise Goodhart's law, either by categorising variants of it, or by looking at how optimising a proxy metric affects the optimisation of an intended goal. In this work, we alleviate the simplifying independence assumption, made in previous works, and the assumption on the learning paradigm made in most of them, to study the effect of the coupling between the proxy metric and the intended goal on Goodhart's law. Our results show that in the case of light tailed goal and light tailed discrepancy, dependence does not change the nature of Goodhart's effect. However, in the light tailed goal and heavy tailed discrepancy case, we exhibit an example where over-optimisation occurs at a rate inversely proportional to the heavy tailedness of the discrepancy between the goal and the metric. %