Categorizing Variants of Goodhart's Law

TL;DR

This paper clarifies different failure modes of Goodhart's Law, emphasizing their relevance in AI and policy, and aims to improve understanding of these optimization failures across various fields.

Contribution

It expands on Garrabrant's identification of four mechanisms behind Goodhart's Law, providing clearer distinctions and explanations of how these failures occur.

Findings

Identifies and clarifies four distinct mechanisms of Goodhart's Law

Highlights the importance of understanding these failure modes in AI and policy

Provides a framework for better diagnosing optimization failures

Abstract

There are several distinct failure modes for overoptimization of systems on the basis of metrics. This occurs when a metric which can be used to improve a system is used to an extent that further optimization is ineffective or harmful, and is sometimes termed Goodhart's Law. This class of failure is often poorly understood, partly because terminology for discussing them is ambiguous, and partly because discussion using this ambiguous terminology ignores distinctions between different failure modes of this general type. This paper expands on an earlier discussion by Garrabrant, which notes there are "(at least) four different mechanisms" that relate to Goodhart's Law. This paper is intended to explore these mechanisms further, and specify more clearly how they occur. This discussion should be helpful in better understanding these types of failures in economic regulation, in public policy, in machine learning, and in Artificial Intelligence alignment. The importance of Goodhart effects depends on the amount of power directed towards optimizing the proxy, and so the increased optimization power offered by artificial intelligence makes it especially critical for that field.

Equations

$$M=G+normal(\mu ,{\sigma }^2)$$

$$M=G(s_i)+G^{\prime }(s_i)$$

$$G=\{\begin{array}{ll}M+x,& \text{where}M<=a\\ M+y,& \text{where}M>a\end{array}$$

$$\begin{array}{r}Metric\sim normal(X,{\sigma }_m^2)\\ Goal\sim normal(X,{\sigma }_g^2)\end{array}$$

$$Metric\sim normal(X+Goal,{\sigma }_m^2)$$

$$Goal\sim normal(X+Metric,{\sigma }_g^2)$$

$$\begin{array}{r}{M}_R=G_R+X\\ M_A=G_A\cdot X\end{array}$$

$$G_{A_R}=G_{A_0}+M_R$$

Full text

Categorizing Variants of Goodhart’s Law

David Manheim Scott Garrabrant

davidmanheim@gmail.com scott@intelligence.org

There are several distinct failure modes for overoptimization of systems on the basis of metrics. This occurs when a metric which can be used to improve a system is used to such an extent that further optimization is ineffective or harmful, and is sometimes termed Goodhart’s Law111As a historical note, Goodhart’s Law [1] as originally formulated states that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” This has been interpreted and explained more widely, perhaps to the point where it is ambiguous what the term means. Other closely related formulations, such as Campbell’s law (which arguably has scholarly precedence[3]) and the Lucas critique, were also initially specific, and their interpretation has also been expanded greatly. Lastly, the Cobra Effect and perverse incentives are often closely related to these failures, and the different effects interact. Because none of the terms were laid out formally, the categories proposed do not match what was originally discussed. A separate forthcoming paper intends to address the relationship between those formulations and the categories more formally explained here.. This class of failure is often poorly understood, partly because terminology for discussing them is ambiguous, and partly because discussion using this ambiguous terminology ignores distinctions between different failure modes of this general type.

This paper expands on an earlier discussion by Garrabrant [2], which notes there are “(at least) four different mechanisms” that relate to Goodhart’s Law. This paper is intended to explore these mechanisms further, and specify more clearly how they occur. This discussion should be helpful in better understanding these types of failures in economic regulation, in public policy, in machine learning, and in artificial intelligence alignment[4]. The importance of Goodhart effects depends on the amount of power directed towards optimizing the proxy, and so the increased optimizationpower offered by artificial intelligence makes it especially critical for that field.

Varieties of Goodhart-like Phenomena

As used in this paper, a Goodhart effect is when optimization causes a collapse of the statistical relationship between a goal which the optimizer intends and the proxy used for that goal. The four categories of Goodhart effects introduced by Garrabrant are

1. Regressional, where selection for an imperfect proxy necessarily also selects for noise,
2. Extremal, where selection for the metric pushes the state distribution into a region where old relationships no longer hold,
3. Causal, where an action on the part of the regulator causes the collapse, and
4. Adversarial, where an agent with different goals than the regulator causes the collapse. These varied forms often occur together, but defining them individually is useful. In doing so, this paper introduces and explains several subcategories which differ in important ways.

To formalize the intuitive description, we consider a system $S$ with a set of possible states $s\in S$. For the initial discussion we focus on a single actor, the regulator, who influences the system by selecting a permissable region of the state-space, $A\subseteq S$. For this discussion, we will use $Goal$ to refer to the true goal of the regulator, which is a mapping from states $G(s)\to\mathbb{R}$ for $s\in S$. Because regulators have incomplete knowledge, they cannot act based on $G(s)$ and instead act only on a proxy $M(s)\to\mathbb{R}$ for $s\in S$222In general, a mapping from $s\to\mathbb{R}$ is a measaure, and if used for decision-making, it is known as a metric. The current presentation assumes a single-dimensional case. Use of multiple metrics and restrictions follows similar dynamics, but for discussing Goodhart effects the single dimensional case is ideal.. For simplicity’s sake, we will consider actions where the regulator chooses some threshold $c$ and the permissible states are defined such that $s\in A$ if $M(s)\geq c$. This creates a selection pressure that allows the first two Goodhart-like effects to occur333This restriction of the available states is one form of selection pressure. There are other forms of selection pressure which can apply, but these are unnecessary for presenting the basic dynamics. For example, we often find that the states are chosen probabilistically and the distribution can be influenced to prefer certain regions. One important such case is evolutionary selection, where the most likely states are generated based on a set of states selected in a previous generation..

1 Regressional Goodhart

Regressional Goodhart

• When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. This is also known as “Tails come apart.” [5]

Simple Model:

[TABLE]

Due to the noise, a point with a large M value will likely have a large G value, but also a large noise value. Thus, when M is large, you can expect G to be predictably smaller than M. Despite the lack of bias, for large values of $c$ the values of G when $M>c$ is expected to be higher than otherwise. While this is the simplest Goodhart effect, it is also the most fundamental: it cannot be avoided. No matter what measure is chosen for optimization, an inexact metric necessarily leads to a divergence between the goal and the metric in the tail.

2 Extremal Goodhart

Extremal Goodhart

• Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the relationship between the proxy and the goal was observed. A form of this occurs occurs in statistics and machine learning as “out of sample prediction.”

Extremal Goodhart can occur in two ways. First, the proxy may have been simplified based on either an insufficient number of observations to uncover the true relationship, or because a simpler measure was preferred. In either case, model insufficiency can lead to extremal Goodhart. Secondly, the measure may be a correct estimator only in certain regions of the space. In regions where the generating process differs, the use of the measure leads to extremal Goodhart.

Extremal Goodhart - Model Insufficiency

• The metric of interest is based on a learned relationship between the goal and the metric which is approximately accurate in the initial region. Selection pressure moves the metric away from the region in which the relationship is most accurate so that the relationship collapses.

Simple Model:

[TABLE]

In the case of model insufficiency, the error is induced by model simplification and inaccuracy, and not a fundamental issue with the inability to build an sufficiently accurate estimator. In the simplest case, model insufficiency involves incomplete understanding of an empirically learned relationship. Even though we assume the relationship is the same within and without the observed range, it is still often more difficult to detect the exact functional relationship based on a limited range. More commonly, model insufficiency occurs because complex systems are approximated using observations of the non-optimized system space.

An example of this phenomenon in public policy is when a relationship is simplified for use as a measure without recognizing how selection pressure applied to the metric makes the simplification problematic. In machine learning this often happens due to underfitting, such as when a relationship is assumed to be a low degree polynomial because higher order polynomial terms are small in the observed region. Selection on the basis of the approximated metric moves towards regions where the higher-order terms are more important, so that use of the machine learning system creates a Goodhart effect.

Extremal Goodhart - Change in Regime

• The proxy M may be related to G differently in different regions. Even if the correct relationship is learned for the observed region, in the region where the proxy takes an extreme value the relationship to the goal may be fundamentally different. Selection on the basis of the proxy moves into such a region.

Simple Model:

[TABLE]

The case of regime change can be due to measurement error so that errors are systemically incorrect in a given region. Alternatively, it can be due to a generating process that differs in defferent regions. An example of the first case is where wind-speed measurements may be systematically biased downwards when the wind-speed exceeds the design tolerances of the instruments. In this case, we expect that the relationship between measured windspeed and wind damage to systematically differ for measured wind-speeds at and above the maximum tolerance. Any selection pressure for the second case would cause entremal Goodhart to occur when a boundary is reached. For example, the relationship between wind speeds and height undergoes a change above the atmospheric boundary layer. Even when the correct relationship is learned for this lower altitude, the relationship changes above it.

In the above cases of regressional and extremal Goodhart, there are issues with selection pressure for even simple systems when proxies are used. These two classes require selection pressure, but do not involve an intervention on the part of the regulator. When a regulator intervenes in the causal sense, it changes the state-space and allows an additional type of error that depends on the causal structure of the intervention.

3 Causal Goodhart

Causal Goodhart

• When the causal path between the proxy and the goal is indirect, intervening can change the relationship between the measure and proxy. If a regulator intervenes to maximize a metric, the causal pathway can change such that the proxy no longer tracks the goal. In such cases, extreme interventions can be less effective than more moderate ones, and further selection or intervention can be counterproductive.

In the earlier cases, the correlational issues found in Goodhart-like effects exist regardless of causal structure. Here, however, a model that ignores causal structure is problematic when the regulator’s actions themselves change the relationships. These take the form of Goodhart-like problems, but can be better discussed in terms of the effect of regulator actions on the causal structure. As a simple example, a regulator attempting to use a windspeed model to build windmills must be careful not to ignore the effect the windmills themselves have on wind-speeds. Building many windmills in a given area can alter the wind-speed in the region enough to invalidate the earlier relationship.

Interestingly, this does not require any uncertainty, nor does it require an incorrect understanding of the relationships, unlike earlier cases. Instead, the effect is induced by the regulator’s action. The exact way in which this occurs can vary, as discussed below.

Three Causal Goodhart Effects

There are a three general cases which lead to causal Goodhart, as well as several others that may appear similar, but are actually regressional or extremal Goodhart effects. The below diagram illustrates three classes of causal Goodhart effects. In each, the regulator attempts to act based on a correlation between the measure and the goal by intervening. The node chosen for intervention is shaded.

Shared Cause Intervention

• The regulator intervenes on a shared cause of the metric and the goal.

Simple Model: The regulator sets the shared cause $X$ to a maximal value. The Metric no longer has a causal relationship to the Goal. The relationship between Metric and Goal now consists entirely of the combination of the error terms between each and $X$.

Shared cause intervention may restrict both the Goal and the Metric to a region favored by the regulator, at the cost of changing the metric’s relationship to the goal in a way that may be counterproductive for further optimization. Suppose, for example, that there are two tests administered to students which are correlated. If a teacher trains skills related to test taking generally, one of the traits which make the students likely to do well on both tests changes. Because of this, the remaining relationship between scores on the two tests is due to other factors, and the new correlation between scores is likely to be lower than the earlier correlation. This does not imply that the overall outcome is worse than without the optimization, but more extreme interventions can be less effective than less extreme ones. For this reason, further selection or intervention on the basis of the metric can be counterproductive.

Intermediary Intervention

• The regulator intervenes on a variable in the causal chain connecting Goal to Metric.

Simple Model: The regulator sets $X$ to a specific value. The relationship between the Goal and Metric now no longer exists; they are independent variables. This does not necessarily affect the Goal at all, but can serve to increase the value of the metric.

Metric Manipulation

• The regulator intervenes to set the Metric, without affecting other nodes. (In this case, it does not matter how the goal was related to X, as indicated in the figure.)

Simple Model: The regulator sets $M$ to a specific value. The relationship between Metric and Goal now no longer holds.

Imagine, to give another example, that a teacher changes test scores or grades444This is assuming the teacher is the regulator - if the teacher is an agent, this is a different case, explored below.. This doesn’t contribute to the goal of learning, it simply changes $M$ so that it is useless (or, if only some scores are changed, less useful) in measuring $G$.

The reason for causal Goodhart effects can either be due to mistaken or missing understanding of causal effects, or simple lack of foresight. As with extremal Goodhart effects, mistakes about the causal connection can be due to insufficient data, or a change in regime. Importantly, the same mistakes can also lead to extremal or (worsened) correlational Goodhart effects if there is selection instead of intervention.

Non-Causal Goodhart Effects in Causal Systems

In the below three cases, the figures show where the regulator mistakes the form of the causal relationship. The dashed lines are the true causal path, and the dotted lines are the assumed ones. If the regulator attempts to select on the basis of the assumed model, it can lead to worsened regressional Goodhart effects or extremal Goodhart effects, but not causal Goodhart effects.

Ignored Shared Cause

• The regulator assumes the relationship between the Goal and Metric is a causal chain, or is direct, but there is instead a shared cause. When ignoring X, the metric is valid as a proxy for the goal.

Simple Model:

[TABLE]

However, because of the conditional independence of Goal and Metric, for any fixed value of $X$ they are uncorrelated. This means that our metric is less valid if there is selection. With selection of a fixed X, the case resembles extremal Goodhart, since the regime changed.

Ignored Intermediary

• The regulator assumes the relationship between the Goal and Metric is direct, but an intermediary exists which creates an additional source of noise.

Without intervention, the causal mistake here adds a term to the error, leading to the earlier cases of regressional and extremal Goodhart effects.

Ignored Additional Cause

• The metric is caused by multiple factors, of which the goal relates to only some. Alternatively, the goal is caused by multiple factors, of which the metric relates to only some. (No Figure provided.)

Simple Model:

[TABLE]

or

[TABLE]

If $X$ has a distribution that matches the distribution of error of the assumed relationship between $X$ and the metric, this leads to worsened regressional Goodhart effects as defined above, because additional noise is due to $X$. In this simple model case the goal is caused by the metric, for example height causing windspeed. More commonly, the metric would be caused by some $Y$, which also causes the Goal. In such a case, the metric might be measured height, where the measurement itself can have error. On the other hand, if $X$ is distributed differently than the assumed distribution of error in the relationship between Goal and Metric, there is a model insufficiency issue. If $X$ contains some nonlinearity, it is a regime change. The earlier examples of extremal Goodhart effects are of these types. (More complex errors can lead to similar mistakes.)

4 Adversarial Goodhart

The discussion of causal relationships is important for a different class of situations where there are multiple actors. There are numerous such dynamics, but they can be classified as happening in one of two cases. First, the actor may have goals which the regulator is unaware of (or insufficiently wary about) and the agent can act independently of the regulator in a way that adversely affects the regulator’s goal. We refer to such cases as adversarial misalignment. Second, the regulator can use incentives to align the agent’s goals and use their actions as a way to optimize, while not acting themself. We refer to these as “Cobra effects.” In all of these cases, which were called “adversarial Goodhart” in the previous work, other agents react in different ways to create Goodhart-like effects following any of the earlier paradigms. We introduce and formulate several specific cases of each.

Adversarial Misalignment Goodhart

• The agent applies selection pressure knowing the regulator will apply different selection pressure on the basis of the metric555This is the case most closely related to Campbell’s law[6], which was originally stated as “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” In this case, the choice of measurement affects the outcome because agents attempt to corrupt the measure..

The adversarial misalignment failure can occur due to the agent creating extremal Goodhart effects, or by exacerbating always-present regressional Goodhart, or due to causal intervention by the agent which changes the effect of the regulator optimization.

Campbell’s Law

• Agents select a metric knowing the choice of regulator metric. Agents can correlate their metric with the regulator’s metric, and select on their metric666They could instead alter causal structure to create the same effect. It seems unclear that this difference is critical to the dynamics considered.. This further reduces the usefulness of selection using the metric for acheiving the original goal.

Simple Model: $G_{R}$, $M_{R}$ are the regulator’s goal and metric. $G_{A}$, $M_{A}$ are the agent’s goal and metric. (The agent selects $M_{A}$ after seeing the regulator choice of metric.)

[TABLE]

Here, the agent selects for values with high $M_{A}$, and the regulator’s later selection then creates a relationship between $X$ and their goal, especially at the extremes. The agent does this by selecting for a metric such that even weak selection on $M_{A}$ hijacks the regulator’s selection on $M_{R}$ to acheive their goal. The agent choice of metric need not be a useful proxy for their goal absent the regulator’s action. In the example given, if $X\thicksim normal(\mu,\sigma^{2})$, the correlation between $G_{A}$ and $M_{A}$ is zero over the full set of states, but becomes positive on the subspace selected by the regulator.

There are clearly further dynamics worth exploring, but this case serves to introduce the issues involved in adversarial conflict over metrics without incentives. When the regulator attempts to regulate via incentives, a new set of cases can occur, which are generally related to what is know as the “Cobra Effect.” [7] This is named after a supposed situation in colonial India where British authorities offered a reward for dead cobras. Instead of hunting cobras, however, some people bred and killed their own cobras to kill in order to receive rewards. This not only failed to achieve the goal, but led to more cobras than before the reward was offered. Because of the (supposedly) historical meaning and popular usage, we differentiate between cobra effects that fit this model, which we will call normal cobra effects, and ones where the agent applies pressure in a non-causal fashion to create Goodhart effects, which we call non-causal cobra effects.

Normal Cobra Effect

• The regulator modifies the agent goal, usually via an incentive, to correlate it with the regulator metric. The agent then acts by changing the observed causal structure due to incompletely aligned goals in a way that creates a Goodhart effect.

Simple Model: The agent finds cause Y (an “Ignored Additional Cause”) which is unobserved or constant in the structure the regulator initially considered, and the agent changes the value of Y in order to maximize the metric. (This is a form of causal Goodhart using metric manipulation.)

[TABLE]

The Cobra effect can also occur via an agent action that creates any of the above-mentioned causal Goodhart effects, namely shared cause, intermediary, or metric manipulation.

Non-Causal Cobra Effect

• The regulator modifies the agent goal to make agent actions aligned with the regulator’s metric. Under selection pressure from the agent, extremal Goodhart effects occur or regressional Goodhart effects are worsened.

Simple Model: $G_{R}$, $M_{R}$ are the regulator’s goal and metric. $G_{A_{0}}$ and $G_{A_{R}}$ are the agent’s goal before and after regulator modification.

[TABLE]

When the agent applies selection pressure, it creates a Goodhart effect on the regulator metric.

Conclusion

This paper represents an attempt to categorize a class of the simple statistical misalignments that occur both in any algorithmic system used for optimization, and in many human systems that rely on metrics for optimization. The dynamics highlighted are hopefully useful to explain many situations of interest in policy design, in machine learning, and in specific questions about AI alignment. In policy, these dynamics are commonly encountered but too-rarely discussed clearly. In machine learning, these errors include extremal Goodhart effects due to using limited data and choosing overly parsimonious models, errors that occur due to myopic consideration of goals, and mistakes that occur when ignoring causality in a system. Finally, in AI alignment, these issues are fundamental to both aligning systems towards a goal, and assuring that the system’s metrics do not have perverse effects once the system begins optimizing for them.