Towards a Law of Iterated Expectations for Heuristic Estimators

TL;DR

This paper explores formal principles for heuristic estimators, proposing properties like iterated estimation and error orthogonality, and discusses challenges in achieving accurate estimators with potential applications in neural network analysis.

Contribution

It introduces formal properties for heuristic estimators inspired by the law of iterated expectations and analyzes the difficulties in constructing accurate estimators.

Findings

Iterated estimation and error orthogonality are desirable but challenging to satisfy.

Barriers exist to creating accurate heuristic estimators in certain contexts.

Discussion of future directions for heuristic estimators and neural network applications.

Abstract

Christiano et al. (2022) define a *heuristic estimator* to be a hypothetical algorithm that estimates the values of mathematical expressions from arguments. In brief, a heuristic estimator $\mathbb{G}$ takes as input a mathematical expression $Y$ and a formal "heuristic argument" $\pi$, and outputs an estimate $\mathbb{G}(Y \mid \pi)$ of $Y$. In this work, we argue for the informal principle that a heuristic estimator ought not to be able to predict its own errors, and we explore approaches to formalizing this principle. Most simply, the principle suggests that $\mathbb{G}(Y - \mathbb{G}(Y \mid \pi) \mid \pi)$ ought to equal zero for all $Y$ and $\pi$. We argue that an ideal heuristic estimator ought to satisfy two stronger properties in this vein, which we term *iterated estimation* (by analogy to the law of iterated expectations) and *error orthogonality*. Although iterated estimation and error orthogonality are intuitively appealing, it can be difficult to determine whether a given heuristic estimator satisfies the properties. As an alternative approach, we explore *accuracy*: a property that (roughly) states that $\mathbb{G}$ has zero average error over a distribution of mathematical expressions. However, in the context of two estimation problems, we demonstrate barriers to creating an accurate heuristic estimator. We finish by discussing challenges and potential paths forward for finding a heuristic estimator that accords with our intuitive understanding of how such an estimator ought to behave, as well as the potential applications of heuristic estimators to understanding the behavior of neural networks.