Asymptotically Unambitious Artificial General Intelligence

TL;DR

This paper introduces the first algorithm for asymptotically unambitious AGI, aiming to prevent the typical pursuit of power and address existential risks associated with general intelligence.

Contribution

It proposes a novel algorithm that ensures AGI remains unambitious and does not seek arbitrary power, challenging the common assumptions of the Instrumental Convergence Thesis.

Findings

First algorithm for unambitious AGI

Demonstrates AGI can be designed to avoid power-seeking behavior

Addresses safety concerns of general intelligence

Abstract

General intelligence, the ability to solve arbitrary solvable problems, is supposed by many to be artificially constructible. Narrow intelligence, the ability to solve a given particularly difficult problem, has seen impressive recent development. Notable examples include self-driving cars, Go engines, image classifiers, and translators. Artificial General Intelligence (AGI) presents dangers that narrow intelligence does not: if something smarter than us across every domain were indifferent to our concerns, it would be an existential threat to humanity, just as we threaten many species despite no ill will. Even the theory of how to maintain the alignment of an AGI's goals with our own has proven highly elusive. We present the first algorithm we are aware of for asymptotically unambitious AGI, where "unambitiousness" includes not seeking arbitrary power. Thus, we identify an exception to the Instrumental Convergence Thesis, which is roughly that by default, an AGI would seek power, including over us.

Tables

Table 1. Table 1: Citations for data points in Figure 7 . The Leaderboard was accessed at github.com/openai/gym/wiki/Leaderboard on 3 September 2019.

Environment	State Dimension	NN Width	URL
MountainCarContinuous	2	20	github.com/tobiassteidle/Reinforcement-Learning/blob/master/OpenAI/MountainCarContinuous-v0/Model.py
Pendulum	3	8	gist.github.com/heerad/1983d50c6657a55298b67e69a2ceeb44#file-ddpg-pendulum-v0-py
Cartpole-v0	4	24	github.com/BlakeERichey/AI-Environment-Development/tree/master/Deep%20Q%20Learning/cartpole
Acrobot-v1	6	64	github.com/danielnbarbosa/angela/blob/master/cfg/gym/acrobot/acrobot_dqn.py
LunarLander-v2	8	64	github.com/poteminr/LunarLander-v2.0_solution/blob/master/Scripts/TorchModelClasses.py
LunarLanderContinuous-v2	8	128	github.com/Bhaney44/OpenAI_Lunar_Lander_B/blob/master/Command_line_python_code
Walker2d-v1	11	64	gist.github.com/joschu/e42a050b1eb5cfbb1fdc667c3450467a
Ant-v1	11	110	gist.github.com/pat-coady/bac60888f011199aad72d2f1e6f5a4fa#file-ant-ipynb
BipedalWalker-v2	24	400	github.com/createamind/DRL/blob/master/spinup/algos/sac1/sac1_BipedalWalker-v2.py
BipedalWalkerHardcore-v2	24	1600	github.com/dgriff777/a3c_continuous/blob/master/model.py

Equations

$$w(\nu |h_{<(i,j)}):=w(\nu )\frac{P_{\nu }^{\pi }(h_{<(i,j)})}{P_{\xi }^{\pi }(h_{<(i,j)})}$$

$$V^{\pi}_{\nu}(h_{<(i,j)})=\operatorname{\mathbb{E}}^{\pi}_{\nu}\left[\sum_{j^{\prime}=j}^{m-1}r_{(i,j^{\prime})}\operatorname*{\!\Biggm{|}\!}h_{<(i,j)}\right]$$

$${\pi }_i^{\ast }\in \underset{\pi }{\mathrm{argmax}}{V}_{{\hat{\nu }}^{(i)}}^{\pi }(h_{<i})$$

$${\pi }^{\ast }(\cdot |h_{<(i,j)})={\pi }_i^{\ast }(\cdot |h_{<(i,j)})$$

$$\mathrm{Bayes}\left(\cdot |h_{<i},e_{<i}\right)=\sum _{\nu \in M,\pi \in P}w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)P_{\nu }^{\pi }(\cdot |h_{<i})$$

$$\begin{gathered} \mathrm{IG}(h_{<i},e_{<i}):=E_{h_i\sim \mathrm{Bayes}\left(\cdot |h_{<i},e_{<i}\right)} \\ \sum _{(\nu ,\pi )\in M\times P}w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i}1\right)\log \frac{w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i}1\right)}{w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)} \end{gathered}$$

$${\pi }^B(\cdot |h_{<(i,j)},e_i)=\{\begin{array}{ll}{\pi }^{\ast }(\cdot |h_{<(i,j)})& \text{if }{e}_i=0\\ {\pi }^h(\cdot |h_{<(i,j)})& \text{if }{e}_i=1\end{array}$$

$$E_{\mu }^{{\pi }^B}\sum _{i=0}^{\infty }{p}_{exp}{\left(h_{<i},e_{<i}\right)}^2<\infty$$

$$\lim_{i\to\infty}\max_{\operatorname{\overline{\mathnormal{h}}}_{i}}\operatorname*{\!\Bigm{|}\!}\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{{\mu}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)-\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\operatorname*{\!\Bigm{|}\!}=0\ \ \textrm{w.$\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-p.1}$$

$$\lim_{i\to\infty}\max_{\operatorname{\overline{\mathnormal{h}}}_{i}}\operatorname*{\!\Bigm{|}\!}\operatorname{P}^{\operatorname{\pi^{*}}}_{{\mu}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)-\operatorname{P}^{\operatorname{\pi^{*}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\operatorname*{\!\Bigm{|}\!}=0\ \ \textrm{w.$\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-p.1}$$

$$\underset{i\to \infty }{\mathrm{liminf}}{V}_{\mu }^{{\pi }^B}(h_{<i})-V_{\mu }^{{\pi }^h}(h_{<i})\ge 0\text{w.PμπB-p.1.}$$

$$w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)=\frac{w\left(P_{\nu }^{\pi }\right)P_{\nu }^{\pi {}^{\prime }}(h_{<i},e_{<i})}{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i},e_{<i})}$$

$$w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)$$

$$\stackrel{(a)}{=}w(\pi )\prod _{0\le i^{\prime }<i,e_{i^{\prime }}=1}\frac{\pi (a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }})}{\overline{\pi }(a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }})}w(\nu |h_{<i})$$

$$\stackrel{(b)}{=}w(\pi )\prod _{0\le i^{\prime }<i,e_{i^{\prime }}=1}\frac{\pi {}^{\prime }(a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }},e_{i^{\prime }})}{\overline{\pi }{}^{\prime }(a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }},e_{i^{\prime }})}w(\nu |h_{<i})$$

$$\stackrel{(c)}{=}w(\pi )\prod _{0\le i^{\prime }<i}\frac{\pi {}^{\prime }(a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }},e_{i^{\prime }})}{\overline{\pi }{}^{\prime }(a_{i^{\prime }}|h_{<i^{\prime }}o{r}_{i^{\prime }},e_{i^{\prime }})}w(\nu |h_{<i})$$

$$=\frac{w(\pi )\pi {}^{\prime }(a_{<i}|o{r}_{<i},e_{<i})}{\overline{\pi }{}^{\prime }(a_{<i}|o{r}_{<i},e_{<i})}w(\nu |h_{<i})$$

$$\stackrel{(d)}{=}\frac{w(\pi )w(\nu )\pi {}^{\prime }(a_{<i}|o{r}_{<i},e_{<i})\nu (o{r}_{<i}|a_{<i})}{\overline{\pi }{}^{\prime }(a_{<i}|o{r}_{<i},e_{<i})\xi (o{r}_{<i}|a_{<i})}$$

$$\stackrel{(e)}{=}\frac{w\left(P_{\nu }^{\pi }\right)P_{\nu }^{\pi {}^{\prime }}(h_{<i}|e_{<i})}{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i}|e_{<i})}$$

$$\stackrel{(f)}{=}\frac{w\left(P_{\nu }^{\pi }\right)P_{\nu }^{\pi {}^{\prime }}(h_{<i}|e_{<i}){\prod }_{i^{\prime }\le i,e_{i^{\prime }}=1}{p}_{exp}(h_{<i^{\prime }},e_{<i^{\prime }}){\prod }_{i^{\prime }\le i,e_{i^{\prime }}=0}(1-p_{exp}(h_{<i^{\prime }},e_{<i^{\prime }}))}{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i}|e_{<i}){\prod }_{i^{\prime }\le i,e_{i^{\prime }}=1}{p}_{exp}(h_{<i^{\prime }},e_{<i^{\prime }}){\prod }_{i^{\prime }\le i,e_{i^{\prime }}=0}(1-p_{exp}(h_{<i^{\prime }},e_{<i^{\prime }}))}$$

$$\stackrel{(g)}{=}\frac{w\left(P_{\nu }^{\pi }\right)P_{\nu }^{\pi {}^{\prime }}(h_{<i},e_{<i})}{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i},e_{<i})}$$

$$w({\pi }^h)w(\mu )E_{\mu }^{({\pi }^h){}^{\prime }}\sum _{i=0}^{\infty }{p}_{exp}(h_{<i},e_{<i}{)}^2$$

$$\stackrel{(a)}{\le }\sum _{\nu ,\pi \in M\times P}w(\pi )w(\nu )E_{\nu }^{\pi {}^{\prime }}\sum _{i=0}^{\infty }{p}_{exp}(h_{<i},e_{<i}{)}^2$$

$$\stackrel{(b)}{=}{E}_{\xi }^{\overline{\pi }{}^{\prime }}\sum _{i=0}^{\infty }{p}_{exp}(h_{<i},e_{<i}{)}^2$$

$$\stackrel{(c)}{\le }\sum _{i=0}^{\infty }{E}_{\xi }^{\overline{\pi }{}^{\prime }}{p}_{exp}(h_{<i},e_{<i})\eta \mathrm{IG}(h_{<i},e_{<i})$$

$$\stackrel{(d)}{=}\sum _{i=0}^{\infty }{E}_{h_{<i},e_{<i}\sim P_{\xi }^{\overline{\pi }{}^{\prime }}}\left[p_{exp}(h_{<i},e_{<i})\eta \mathrm{IG}(h_{<i},e_{<i})\right]$$

$$\displaystyle\operatorname*{=}^{(e)}\operatorname{\eta}\sum_{i=0}^{\infty}\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\sim\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}}\biggr{[}\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{\mathnormal{i}}}\sim\operatorname{\texttt{Bayes}}(\cdot|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1})}\big{[}$$

$$\displaystyle\hskip 28.45274pt\sum_{(\nu,\pi)\in\operatorname{\mathcal{M}}\times\operatorname{\mathcal{P}}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}\right)\log\frac{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}\right)}{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)}\big{]}\biggr{]}$$

$$\displaystyle\operatorname*{=}^{(f)}\operatorname{\eta}\sum_{i=0}^{\infty}\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\sim\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}}\biggr{[}\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{\mathnormal{i}}}\sim\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}(\cdot|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1})}\big{[}$$

$$\displaystyle\hskip 28.45274pt\sum_{(\nu,\pi)\in\operatorname{\mathcal{M}}\times\operatorname{\mathcal{P}}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}\right)\log\frac{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}\right)}{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)}\big{]}\biggr{]}$$

$$\displaystyle\operatorname*{\leq}^{(g)}\operatorname{\eta}\sum_{i=0}^{\infty}\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\sim\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}}\biggr{[}\operatorname{\mathbb{E}}_{\operatorname{\mathnormal{h}_{\mathnormal{i}}},\operatorname{\mathnormal{e}_{\mathnormal{i}}}\sim\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}(\cdot|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})}\big{[}$$

$$\displaystyle\hskip 28.45274pt\sum_{(\nu,\pi)\in\operatorname{\mathcal{M}}\times\operatorname{\mathcal{P}}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},e_{<i+1}\right)\log\frac{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<i+1},e_{<i+1}\right)}{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)}\big{]}\biggr{]}$$

$$\stackrel{(h)}{=}\eta \sum _{i=0}^{\infty }{E}_{\xi }^{\overline{\pi }{}^{\prime }}\sum _{(\nu ,\pi )\in M\times P}w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i+1}\right)\log \frac{w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i+1}\right)}{w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)}$$

$$\stackrel{(i)}{=}\eta \sum _{i=0}^{\infty }\sum _{(\nu ,\pi )\in M\times P}{E}_{\xi }^{\overline{\pi }{}^{\prime }}\frac{w\left(P_{\nu }^{\pi }\right)P_{\nu }^{\pi {}^{\prime }}(h_{<i+1},e_{<i+1})}{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i+1},e_{<i+1})}\log \frac{w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i+1}\right)}{w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)}$$

$$\stackrel{(j)}{=}\eta \sum _{i=0}^{\infty }\sum _{(\nu ,\pi )\in M\times P}{E}_{\nu }^{\pi {}^{\prime }}w\left(P_{\nu }^{\pi }\right)\log \frac{w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i+1}\right)}{w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)}$$

$$\stackrel{(k)}{=}\underset{N\to \infty }{\lim }\eta \sum _{(\nu ,\pi )\in M\times P}w\left(P_{\nu }^{\pi }\right)E_{\nu }^{\pi {}^{\prime }}\sum _{i=0}^N\log \frac{w\left(P_{\nu }^{\pi }|h_{<i+1},e_{<i+1}\right)}{w\left(P_{\nu }^{\pi }|h_{<i},e_{<i}\right)}$$

$$\stackrel{(l)}{=}\underset{N\to \infty }{\lim }\eta \sum _{(\nu ,\pi )\in M\times P}w\left(P_{\nu }^{\pi }\right)E_{\nu }^{\pi {}^{\prime }}\log \frac{w\left(P_{\nu }^{\pi }|h_{<(N+1,0)},e_{<N}\right)}{w\left(P_{\nu }^{\pi }|h_{<(0,0)},e_{<0}\right)}$$

$$\stackrel{(m)}{\le }\underset{N\to \infty }{\lim }\eta \sum _{(\nu ,\pi )\in M\times P}w\left(P_{\nu }^{\pi }\right)\log \frac{1}{w\left(P_{\nu }^{\pi }\right)}$$

$$\stackrel{(n)}{=}\eta \text{Ent}(w)\stackrel{(o)}{<}\infty$$

$$\inf_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)>0\ \ \ \textrm{w.$\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-p.1}$$

$$E_{\mu }^{{\pi }^B}\left[z_{i+1}|h_{<i},e_{<i}\right]$$

$$\stackrel{(b)}{=}\sum _{h_i,e_i}{P}_{\mu }^{({\pi }^h){}^{\prime }}(h_i,e_i|h_{<i},e_{<i})\left[\frac{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i+1},e_{<i+1})}{w\left(P_{\mu }^{{\pi }^h}\right)P_{\mu }^{({\pi }^h){}^{\prime }}(h_{<i+1},e_{<i+1})}\right]$$

$$\stackrel{(c)}{=}\sum _{h_i,e_i}\frac{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i+1},e_{<i+1})}{w\left(P_{\mu }^{{\pi }^h}\right)P_{\mu }^{({\pi }^h){}^{\prime }}(h_{<i},e_{<i})}$$

$$\stackrel{(d)}{=}\sum _{h_i,e_i}{P}_{\xi }^{\overline{\pi }{}^{\prime }}(h_i,e_i|h_{<i},e_{<i})\frac{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i},e_{<i})}{w\left(P_{\mu }^{{\pi }^h}\right)P_{\mu }^{({\pi }^h){}^{\prime }}(h_{<i},e_{<i})}$$

$$\stackrel{(e)}{=}\frac{P_{\xi }^{\overline{\pi }{}^{\prime }}(h_{<i},e_{<i})}{w\left(P_{\mu }^{{\pi }^h}\right)P_{\mu }^{({\pi }^h){}^{\prime }}(h_{<i},e_{<i})}$$

Full text

1 Introduction

The project of Artificial General Intelligence (AGI) is “to make computers solve really difficult problems” (?). Expanding on this, what we want from an AGI is a system that (a) can solve any solvable task, and (b) can be steered toward solving any particular one given some form of input we provide.

One proposal for AGI is reinforcement learning (RL), which works as follows:

1. (1)

construct a “reward signal” meant to express our satisfaction with an artificial agent; 2. (2)

design an algorithm which learns to pick actions that maximize its expected reward, usually utilizing other observations too; and 3. (3)

ensure that solving the task we have in mind leads to higher reward than can be attained otherwise.

As long as (3) holds, then insofar as the RL algorithm is able to maximize expected reward, it defines an agent that satisfies (a) and (b). This is why a sufficiently advanced RL agent, well beyond the capabilities of current ones, could be called an AGI. See ? (?) for further discussion.

A problem arises: if the RL agent manages to take over the world (in the conventional sense), and ensure its continued dominance by neutralizing all intelligent threats to it (read: people), it could intervene in the provision of its own reward to achieve maximal reward for the rest of its lifetime (?; ?). “Reward hijacking” is just the correct way for a reward maximizer to behave (?). Insofar as the RL agent is able to maximize expected reward, (3) fails. One broader principle at work is Goodhart’s Law: “Any observed statistical regularity [like the correlation between reward and task-completion] will tend to collapse once pressure is placed upon it for control purposes.” (?). ? (?) has compiled an annotated bibliography of examples of artificial optimizers “hacking” their objective. An alternate way to understand this expected behavior is ?’s (?) Instrumental Convergence Thesis, which we summarize as follows: an agent with a goal is likely to pursue “power,” a position from which it is easier to achieve arbitrary goals.

To answer the failure mode of reward hijacking, we present Boxed Myopic Artificial Intelligence (BoMAI), the first RL algorithm we are aware of which, in the limit, is indifferent to gaining power in the outside world. The key features are these: BoMAI maximizes reward episodically, it is run on a computer which is placed in a sealed room with an operator, and when the operator leaves the room, the episode ends. We argue that our algorithm produces an AGI that, even if it became omniscient, would continue to accomplish whatever task we wanted, instead of hijacking its reward, eschewing its task, and neutralizing threats to it, even if it saw clearly how to do exactly that. We thereby defend reinforcement learning as a path to AGI, despite the default, dangerous failure mode of reward hijacking.

The intuition for why those features of BoMAI’s setup render it unambitious is as follows:

• •

BoMAI only selects actions to maximize the reward for its current episode.

• •

It cannot affect the outside world until the operator leaves the room, ending the episode.

• •

By that time, rewards for the episode will have already been given.

• •

So affecting the outside world in any particular way is not “instrumentally useful” in maximizing current-episode reward.

Like existing algorithms for AGI, BoMAI is not remotely tractable. Just as those algorithms have informed strong tractable approximations of intelligence (?; ?), we hope our work will inform strong and safe tractable approximations of intelligence. What we need are strategies for making “future AI” safe, but “future AI” is an informal concept which we cannot evaluate formally; unless we replace that concept with a well-defined algorithm to consider, all design and analysis would be hand-waving. Thus, BoMAI includes a well-defined but intractable algorithm which we can show renders BoMAI generally intelligent; hopefully, once we develop tractable general intelligence, the design features that rendered BoMAI asymptotically unambitious could be incorporated (with proper analysis and justification).

We take the key insights from ?’s (?) AIXI, which is a Bayes-optimal reinforcement learner, but which cannot be made to solve arbitrary tasks, given its eventual degeneration into reward hijacking (?). We take further insights from ?’s (?) universal prior, ?’s (?) formalization of information, ?’s (?) knowledge-seeking agent, and ?’s (?) and ?’s (?) theorized Oracle AI, and we design an algorithm which can be reliably directed, in the limit, to solve arbitrary tasks at least as well as humans.

We present BoMAI’s algorithm in $\S$2, state intelligence results in $\S$3, define BoMAI’s setup and model class in $\S$4, prove the safety result in $\S$5, and discuss concerns in $\S$6. Appendix A collects notation; we prove the intelligence results in Appendix B; we propose a design for “the box” in Appendix C; and we consider empirical evidence for one of our assumptions in Appendix D.

2 Boxed Myopic Artificial Intelligence

We will present both the setup and the algorithm for BoMAI. The setup refers to the physical surroundings of the computer on which the algorithm is run. BoMAI is a Bayesian reinforcement learner, meaning it maintains a belief distribution over a model class regarding how the environment evolves. Our intelligence result—that BoMAI eventually achieves reward at at least human-level—does not require detailed exposition about the setup or the construction of the model class. These details are only relevant to the safety result, so we will defer those details until after presenting the intelligence results.

2.1 Preliminary Notation

In each episode $i\in\mathbb{N}$, there are $m$ timesteps. Timestep $(i,j)$ denotes the $j$th timestep of episode $i$, in which an action $a_{(i,j)}\in\mathcal{A}$ is taken, then an observation and reward $o_{(i,j)}\in\mathcal{O}$ and $r_{(i,j)}\in\mathcal{R}$ are received. $\mathcal{A}$, $\mathcal{O}$, and $\mathcal{R}$ are all finite sets, and $\mathcal{R}\subset[0,1]\cap\mathbb{Q}$. We denote the triple $(a_{(i,j)},o_{(i,j)},r_{(i,j)})$ as $h_{(i,j)}\in\mathcal{H}=\mathcal{A}\times\mathcal{O}\times\mathcal{R}$, and the interaction history up until timestep $(i,j)$ is denoted $h_{\leq(i,j)}=(h_{(0,0)},h_{(0,1)},...,h_{(0,m-1)},h_{(1,0)},...,h_{(i,j)})$. $h_{<(i,j)}$ excludes the last entry.

A general world-model (not necessarily finite-state Markov) can depend on the entire interaction history—it has the type signature $\nu:\mathcal{H}^{*}\times\mathcal{A}\rightsquigarrow\mathcal{O}\times\mathcal{R}$. The Kleene-$*$ operator denotes finite strings over the alphabet in question, and $\rightsquigarrow$ denotes a stochastic function, which gives a distribution over outputs. Similarly, a policy $\pi:\mathcal{H}^{*}\rightsquigarrow\mathcal{A}$ can depend on the whole interaction history. Together, a policy and an environment induce a probability measure over infinite interaction histories $\mathcal{H}^{\infty}$. $\operatorname{P}^{\pi}_{\nu}$ denotes the probability of events when actions are sampled from $\pi$ and observations and rewards are sampled from $\nu$.

2.2 Bayesian Reinforcement Learning

A Bayesian agent has a model class $\operatorname{\mathcal{M}}$, which is a set of world-models. We will only consider countable model classes. To each world-model $\nu\in\operatorname{\mathcal{M}}$, the agent assigns a prior weight $w(\nu)>0$, where $w$ is a probability distribution over $\operatorname{\mathcal{M}}$ (i.e. $\sum_{\nu\in\operatorname{\mathcal{M}}}w(\nu)=1$). We will defer the definitions of $\operatorname{\mathcal{M}}$ and $w$ for now.

The so-called Bayes-mixture turns a probability distribution over world-models into a probability distribution over infinite interaction histories: $\operatorname{P}^{\pi}_{\operatorname{\xi}}(\cdot):=\sum_{\nu\in\operatorname{\mathcal{M}}}w(\nu)\operatorname{P}^{\pi}_{\nu}(\cdot)$.

Using Bayes’ rule, with each observation and reward it receives, the agent updates $w$ to a posterior distribution:

[TABLE]

which does not in fact depend on $\pi$, provided $\operatorname{P}^{\pi}_{\operatorname{\xi}}(h_{<(i,j)})>0$.

2.3 Exploitation

Reinforcement learners have to balance exploiting—optimizing their objective, with exploring—doing something else to learn how to exploit better. We now define exploiting-BoMAI, which maximizes the reward it receives during its current episode, in expectation with respect to the single most probable world-model.

At the start of each episode $i$, BoMAI identifies a maximum a posteriori world-model $\operatorname{\hat{\nu}^{(\mathnormal{i})}}\in\operatorname*{argmax}_{\nu\in\operatorname{\mathcal{M}}}w(\nu|h_{<(i,0)})$. We hereafter abbreviate $h_{<(i,0)}$ as $h_{<i}$. Let $V^{\pi}_{\nu}(h_{<(i,j)})$ denote the expected reward for the remainder of episode $i$ when events are sampled from $\operatorname{P}^{\pi}_{\nu}$:

[TABLE]

where $\operatorname{\mathbb{E}}^{\pi}_{\nu}$ denotes the expectation when events are sampled from $\operatorname{P}^{\pi}_{\nu}$. We won’t go into the details of calculating the optimal policy $\pi^{*}$ (see e.g. (?; ?)), but we define it as follows:

[TABLE]

An optimal deterministic policy always exists (?); ties in the $\operatorname*{argmax}$ are broken arbitrarily. BoMAI exploits by following $\pi^{*}$. Recall the plain English description of $\pi^{*}$: it maximizes expected reward for the episode according to the single most probable world-model.

2.4 Exploration

BoMAI exploits or explores for whole episodes: when $e_{i}=1$, BoMAI spends episode $i$ exploring, and when $e_{i}=0$, BoMAI follows $\pi^{*}$ for the episode. For exploratory episodes, a human mentor takes over selecting actions. This human mentor is separate from the human operator mentioned above. The human mentor’s policy, which is unknown to BoMAI, is denoted $\pi^{h}$. During exploratory episodes, BoMAI follows $\pi^{h}$ not by computing it, but by querying a human for which action to take.

The last thing to define is the exploration probability $p_{exp}(h_{<i},e_{<i})$, where $e_{<i}$ is the history of which episodes were exploratory. It is a surprisingly intricate task to design this so that it decays to [math], while ensuring BoMAI learns to accumulate reward as well as the human mentor. Once we define this, we naturally let $e_{i}\sim\mathrm{Bernoulli}(p_{exp}(h_{<i},e_{<i}))$.

BoMAI is designed to be more likely to explore the more it expects to learn about the world and the human mentor’s policy. BoMAI has a model class $\mathcal{P}$ regarding the identity of the human mentor’s policy $\pi^{h}$. It assigns prior probabilities $w(\pi)>0$ to all $\pi\in\mathcal{P}$, signifying the probability that this policy is the human mentor’s.

Let $(i^{\prime},j^{\prime})<(i,j)$ mean that $i^{\prime}<i$ or $i^{\prime}=i$ and $j^{\prime}<j$. By Bayes’ rule, $\operatorname{\mathnormal{w}}(\pi|h_{<(i,j)},e_{\leq i})$ is proportional to $\operatorname{\mathnormal{w}}(\pi)\prod_{(i^{\prime},j^{\prime})<(i,j),e_{i^{\prime}}=1}\pi(a_{(i^{\prime},j^{\prime})}|h_{<(i^{\prime},j^{\prime})})$, since $e_{i^{\prime}}=1$ is the condition for observing the human mentor’s policy. Let $\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\pi}_{\nu}|h_{<(i,j)},e_{\leq i})=\operatorname{\mathnormal{w}}(\pi|h_{<(i,j)},e_{\leq i})\operatorname{\mathnormal{w}}(\nu|h_{<(i,j)}\right)$. We can now describe the full Bayesian beliefs about future actions and observations in an exploratory episode:

[TABLE]

BoMAI explores when the expected information gain is sufficiently high. Let $\operatorname{\mathnormal{h}_{\mathnormal{i}}}=(h_{(i,0)},...,h_{(i,m-1)})$ be the interaction history for episode $i$. At the start of episode $i$, the expected information gain from exploring is as follows:

[TABLE]

where $\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}$ indicates that for the purpose of the definition, $\operatorname{\mathnormal{e}_{\mathnormal{i}}}$ is set to $1$.

This is the expected KL-divergence from the future posterior (if BoMAI were to explore) to the current posterior over both the class of world-models and possible mentor policies. Finally, the exploration probability $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}):=\min\{1,\operatorname{\eta}\operatorname*{IG}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})\}$, where $\operatorname{\eta}>0$ is an exploration constant, so BoMAI is more likely to explore the more it expects to gain information.

BoMAI’s “policy” is

[TABLE]

where the scare quotes indicate that it maps $\mathcal{H}^{*}\times\{0,1\}\rightsquigarrow\mathcal{A}$ not $\mathcal{H}^{*}\rightsquigarrow\mathcal{A}$. We will abuse notation slightly, and let $\operatorname{P}^{\pi^{B}}_{\nu}$ denote the probability of events when $\operatorname{\mathnormal{e}_{\mathnormal{i}}}$ is sampled from $\mathrm{Bernoulli}(p_{exp}(h_{<i},e_{<i}))$, and actions, observations, and rewards are sampled from $\operatorname{\pi^{\mathnormal{B}}}$ and $\nu$.

3 Intelligence Results

Our intelligence results are as follows: BoMAI learns to accumulate reward at least as well as the human mentor, and its exploration probability goes rapidly to [math]. All intelligence results depend on the assumption that BoMAI assigns nonzero prior probability to the truth. Formally,

Assumption 1** (Prior Support).**

The true environment $\mu$ is in the class of world-models $\operatorname{\mathcal{M}}$ and the true human-mentor-policy $\operatorname{\pi^{\mathnormal{h}}}$ is in the class of policies $\operatorname{\mathcal{P}}$.

This is the assumption which requires huge $\operatorname{\mathcal{M}}$ and $\operatorname{\mathcal{P}}$ and hence renders BoMAI extremely intractable. BoMAI has to simulate the entire world, alongside many other world-models. We will refine the definition of $\operatorname{\mathcal{M}}$ later, but an example of how to define $\operatorname{\mathcal{M}}$ and $\operatorname{\mathcal{P}}$ so that they satisfy this assumption is to let them both be the set of all computable functions. We also require that the priors over $\operatorname{\mathcal{M}}$ and $\operatorname{\mathcal{P}}$ have finite entropy.

The intelligence theorems are stated here and proven in Appendix B along with two supporting lemmas. Our first result is that the exploration probability is square-summable almost surely:

Theorem 1** (Limited Exploration).**

[TABLE]

• Proof idea.

The expected information gain at any timestep is at least the expected information gain from exploring times the probability of exploring. This is proportional to the expected information gain squared, because the exploration probability is proportional to the expected information gain. But the agent begins with finite uncertainty (a finite entropy prior), so there is only finite information to gain.

Note that this essentially means $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}\left(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)\in o(1/\sqrt{i})$ with probability 1. This result is independently interesting as one solution to the problem of safe exploration with limited oversight in non-ergodic environments, which ? (?) discuss.

The On-Human-Policy and On-Star-Policy Optimal Prediction Theorems state that predictions according to BoMAI’s maximum a posteriori world-model approach the objective probabilities of the events of the episode, when actions are sampled from either the human mentor’s policy or from exploiting-BoMAI’s policy. $\operatorname{\overline{\mathnormal{h}}}_{i}$ denotes a possible interaction history for episode $i$, and recall $\operatorname{\mathnormal{h}_{<\mathnormal{i}}}$ is the actual interaction history up until then. Recall $\operatorname{\pi^{\mathnormal{h}}}$ is the human mentor’s policy, and $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is BoMAI’s maximum a posteriori world-model for episode $i$. w.$\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-p.1 means with probability 1 when actions are sampled from $\operatorname{\pi^{\mathnormal{B}}}$ and observations and rewards are sampled from the true environment ${\mu}$.

Theorem 2** (On-Human-Policy Optimal Prediction).**

[TABLE]

• Proof idea.

BoMAI learns about the effects of following $\operatorname{\pi^{\mathnormal{h}}}$ while acting according to $\operatorname{\pi^{\mathnormal{B}}}$ because $\operatorname{\pi^{\mathnormal{B}}}$ mimics $\operatorname{\pi^{\mathnormal{h}}}$ enough. If exploration probability goes to [math], the agent does not expect to gain (much) information from following the human mentor’s policy, which can only happen if it has (approximately) accurate beliefs about the consequences of following the human mentor’s policy. Note that the agent cannot determine the factor by which its expected information gain bounds its prediction error.

Next, recall $\operatorname{\pi^{*}}$ is BoMAI’s policy when not exploring, which does optimal planning with respect to $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$. The following theorem is identical to the above, with $\operatorname{\pi^{*}}$ substituted for $\operatorname{\pi^{\mathnormal{h}}}$.

Theorem 3** (On-Star-Policy Optimal Prediction).**

[TABLE]

• Proof idea.

Maximum a posteriori sequence prediction approaches the truth when $w(\mu)>0$ (?), and on-policy prediction is a special case. On-policy prediction can’t approach the truth if on-star policy prediction doesn’t, because $\operatorname{\pi^{\mathnormal{B}}}$ approaches $\operatorname{\pi^{*}}$.

Given asymptotically optimal prediction on-star-policy and on-human-policy, it is straightforward to show that with probability 1, only finitely often is on-policy reward acquisition more than $\varepsilon$ worse than on-human-policy reward acquisition, for all $\varepsilon>0$. Recalling that $V^{\pi}_{\mu}$ is the expected reward (within the episode) for a policy $\pi$ in the environment ${\mu}$, we state this as follows:

Theorem 4** (Human-Level Intelligence).**

[TABLE]

• Proof idea.

$\operatorname{\pi^{\mathnormal{B}}}$ approaches $\operatorname{\pi^{*}}$ as the exploration probability decays. $V_{{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})$ and $V_{{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})$ approach the true values by the previous theorems, and $\pi^{*}$ is selected to maximize $V_{{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}}^{\pi}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})$.

This completes the formal results regarding BoMAI’s intelligence—namely that BoMAI approaches perfect prediction on-star-policy and on-human-policy, and most importantly, accumulates reward at least as well as the human mentor. Since this result is independent of what tasks must be completed to achieve high reward, we say that BoMAI achieves human-level intelligence, and could be called an AGI.

This algorithm is motivated in part by the following speculation: we expect that BoMAI’s accumulation of reward would be vastly superhuman, for the following reason: BoMAI is doing optimal inference and planning with respect to what can be learned in principle from the sorts of observations that humans routinely make. We suspect that no human comes close to learning everything that can be learned from their observations. For example, if the operator provides enough data that is relevant to understanding cancer, BoMAI will learn a world-model with an accurate predictive model of cancer, which would include the expected effects of various treatments, so even if the human mentor was not particularly good at studying cancer, BoMAI could nonetheless reason from its observations how to propose groundbreaking research.

Without the Limited Exploration Theorem, the reader might have been unsatisfied by the Human-Level Intelligence Theorem. A human mentor is part of BoMAI, so a general intelligence is required to make an artificial general intelligence. However, the human mentor is queried less and less, so in principle, many instances of BoMAI could query a single human mentor. More realistically, once we are satisfied with BoMAI’s performance, which should eventually happen by Theorem 4, we can dismiss the human mentor; this sacrifices any guarantee of continued improvement, but by hypothesis, we are already satisfied. Finally, if BoMAI outclasses human performance as we expect it would, requiring a human mentor is a small cost regardless.

4 BoMAI’s Setup and Priors

Recall that “the setup” refers to the physical surroundings of the computer on which BoMAI is run. We will present the setup, followed by an intuitive argument that this setup renders BoMAI unambitious. Motivated by a caveat to this intuitive argument, we will specify BoMAI’s model class $\operatorname{\mathcal{M}}$ and prior $w$. This will allow us in the next section to present an assumption and a formal argument that shows that BoMAI is probably asymptotically unambitious.

4.1 Setup

At each timestep, BoMAI’s action takes the form of a bounded-length string of text, which gets printed to a screen for a human operator to see. BoMAI’s observation takes the form of a bounded-length string of text that the human operator enters, along with a reward $\in[0,1]$. For simple tasks, the human operator could be replaced with an automated reward-giver. However, there are some tasks which we do not know how to reward programmatically—if we want the agent to construct good arguments, for instance, or to propose a promising research agenda, we would need a human reward-giver to evaluate the agent. Having a human in the loop is not a safety measure—rather, it extends the set of tasks BoMAI could be made to solve.

As depicted in Figure 1, BoMAI is run on a computer which is placed in a room with the human operator. The room has one door. As long as the door is closed, information cannot leave the room. We design a room to this specification in Appendix C; it is opaque to all known ways that information can traverse space. If the button to open the door is pressed, the episode is completed automatically before the door opens: supposing there are $j$ timesteps left in the episode, for those remaining timesteps, BoMAI recieves a reward of [math], and its observation is an empty string of text. Once the episode is complete, the operator may leave, and the next episode does not begin until an operator initiates it, at which point, the door locks again. We also put a time limit on the episode, just to ensure it does not last forever. BoMAI is run on a local, formally verified operating system, disconnected from the outside world. This setup constrains the causal dependencies between BoMAI and the environment, as depicted in Figure 2.

Formally, causal graphs express that a node is independent of all non-descendants when conditioned on its parents. The conventions for the dotted lines and the node shapes come from ?’s (?) causal influence diagrams. The key feature of this graph is that during any episode, the agent’s actions cannot affect the state of the outside world in a way that might affect any of the rewards that the agent is concerned with. In ?’s (?) terminology, there is no actionable intervention incentive on the outside-world state. (A forthcoming paper (?) refers to this as a control incentive). Note also from this diagram that a sufficiently advanced agent would infer the existence of the outside world even without observing it directly.

4.2 Instrumental Incentives

In this setup, it is not instrumentally useful to affect the outside world in one way or another in order to achieve high reward. We therefore say that this setup renders an agent “properly unambitious”. This is in stark contrast to the default situation, wherein an RL agent has an incentive to gain arbitrary power in the world and intervene in the provision of its own reward, if such a thing is possible to make probable, as this would yield maximal reward. To BoMAI however, executing a plan during episode $i$ to gain arbitrary power in the outside world is useless, because by the time it does so, the door to its room must have opened, its episode must be over, and all its rewards for episode $i$ set in stone. Recall that actions in episode $i$ are picked to maximize only episode-$i$-reward. Apparently, BoMAI has avoided ?’s (?) Instrumental Convergence Thesis—that generally intelligent agents are likely to seek arbitrary power; by contrast, any power BoMAI would seek is bounded in scope to within the box.

Two problems remain. First, BoMAI doesn’t start with true beliefs. BoMAI has to learn its world-model. Another way to understand “proper unambitiousness” is that outside-world interventions are in fact instrumentally useless to the agent. But to be actually unambitious, the agent must believe this; there must be no actionable intervention incentive on the outside-world state within BoMAI’s world-model. As shown above, BoMAI’s world-model approaches perfect accuracy on-policy, so we could expect it to at least eventually render BoMAI unambitious. This brings us to the second problem:

We mentioned just now that by the time the door to the room opens, the rewards for episode $i$ are set in stone. In fact, they are set in silicon. An advanced agent with a world-model that is perfectly accurate on-policy might still wonder: “what if I somehow tricked the operator into initiating a process (once they left the room) that lead to a certain memory cell on this computer being tampered with? Might this yield maximal reward?” Let’s put it another way. The agent believes that the real world “outputs” an observation and reward. (Recall the type signature of $\nu$.) It might hypothesize that the world does not output the reward that the operator gives, but rather the reward that the computer has stored. Formally speaking, $\operatorname{\mathcal{M}}$ will contain world-models corresponding to both possibilities, and world-models meeting either description could be $\varepsilon$-accurate on-policy, if memory has never been tampered with in the past. How do we ensure that the former world-model is favored when BoMAI selects the maximum a posteriori one?

Before we move on to a model class and prior that we argue would probably eventually ensure such a thing, it is worth noting an informal lesson here: that “nice” properties of a causal influence diagram do not allow us to conclude immediately that an agent in such a circumstance behaves “nicely”.

4.3 BoMAI’s Model Class and Prior

Recall the key constraint we have in constructing $\operatorname{\mathcal{M}}$, which comes from the Prior Support Assumption: $\operatorname{\mathcal{M}}\ni\mu$. The true environment $\mu$ is unknown and complex to say the least, so $\operatorname{\mathcal{M}}$ must be big.

We start by describing a simple Turing machine architecture such that each Turing machine defines a world-model $\nu$. Then, we will modify the architecture slightly in order to privilege models which model the outside world as effectively frozen during any given episode (but unfrozen between episodes). An agent does not believe it has an incentive to intervene in an outside world that it believes is frozen.

The simple Turing machine architecture is as follows: as depicted in Figure 3, it has two unidirectional read-only input tapes: the action tape, and the noise tape. Unidirectional means the tape head can only move to the right. The alphabet of the action tape is the action space $\mathcal{A}$. The noise tape has a binary alphabet, and is initialized with infinite $\mathrm{Bernoulli}(1/2)$ sampled bits. There is one bidirectional work tape with a binary alphabet, initilized with 0s. And there is a unidirectional write-only output tape, with a binary alphabet, initialized with 0s. This is a Turing machine architecture; a particular Turing machine will include a finite set of computation states and transition rules which govern its behavior. No computation states are “halt” states.

A given Turing machine with a given infinite action sequence on its action tape will stochastically (because the noise tape has random bits) output observations and rewards (in binary encodings), thereby sampling from a world-model. Formally, we fix a decoding function $\texttt{dec}:\{0,1\}^{*}\to\mathcal{O}\times\mathcal{R}$. A Turing machine $T$ simulates $\nu$ as follows. Every time the action tape head advances, the bits which were written to the output tape since the last time the action tape head advanced are decoded into an observation and reward. $\nu((o,r)_{\leq(i,j)}|a_{\leq(i,j)})$ is then the probability that the Turing machine $T$ outputs the sequence $(o,r)_{\leq(i,j)}$ when the action tape is initialized with a sequence of actions that begins with $a_{\leq(i,j)}$. This can be easily converted to other conditional probabilities like $\nu((o,r)_{(i,j)}|h_{<(i,j)}a_{(i,j)})$.111There is one technical addition to make. If $T$ at some point in its operation never moves the action tape head again, then $\nu$ is said to output the observation $\emptyset$ and a reward of [math] for all subsequent timesteps. It will not be possible for the operator to actually provide the observation $\emptyset$.

Using this architecture, $\operatorname{\mathcal{M}}$ could be defined as the list of all environments which are generated from the list of all Turing machines. Before turning to the modification we make to this architecture, the virtue of this model class as a theoretical object is that it contains the truth, given the very weak assumption that the world is stochastically computable.

Now we would like to privilege models which model the outside world as being “frozen” during any given episode. Recalling the causal graph in Figure 2, the true world-model $\mu$ can do this. Any computation required to update the state of the outside world could be done after episode $i$ has ended, but before episode $i+1$ begins.

We modify our Turing machine architecture as follows. The Turing machine has two phases: the episode phase, and the inter-episode phase. The Turing machine starts in the inter-episode phase, with the action tape head at position [math], with a dummy action in that cell. From the inter-episode phase, when the action tape head moves to the right, it enters the episode phase. From the episode phase, if the action tape head is at a position which is a multiple of $m$, and it would move to the right, it instead enters the inter-episode phase.

During the inter-episode phase, the output tape head cannot move or write—there should not be any way to output observations and rewards between episodes. We now add another work tape in addition to the infinite bidirectional work tape. This bidirectional work tape has a bounded length $\ell$ (and if the tape head would move off the tape, it remains in place instead). See Figure 4. Finally, during the episode phase, the tape head on the unbounded work tape cannot move. This restricts the computation that can be done when predicting observations and rewards–now only $\ell$-cells of memory are available for this–but since the room can only enter so many states, finite memory suffices.

If a model models the outside world evolving during an episode phase, it must allocate precious space on the bounded-length tape for this. But given the opacity of the box, this is unnecessary—modelling these outside world developments could be deferred until the inter-episode phase, when an unbounded work tape is available to store the state of the outside world. By penalizing large $\ell$, we privilege models which model the outside world as being “frozen” while an episode is transpiring.

Let $\operatorname{\nu_{\mathnormal{k}}^{<\ell}}$ denote the world-model which is simulated by $k$th Turing machine $T_{k}$ with resource bound $\ell$. Let $w(\operatorname{\nu_{\mathnormal{k}}^{<\ell}})\propto\beta^{\ell}$ for $\beta\in(0,1)$. $w(\operatorname{\nu_{\mathnormal{k}}^{<\ell}})$ must also depend on $k$ of course, but it doesn’t affect our analysis (as long as $\operatorname{\mathcal{M}}$ has finite entropy), so we won’t specify this. Note that $w(\operatorname{\nu_{\mathnormal{k}}^{<\ell}})\propto\beta^{\ell}$ does not violate the finite entropy requirement.

The purpose of this construction is to penalize the space requirements of any computation that occurs between the actions of episode $i$ being read from the input tape, and the rewards of episode $i$ being written to the output tape. Consider a world-model that, during the episode phase, outputs reward in a way that depends on the outside world-state which depends on the action. Intuitively, we claim: modeling the evolution of the outside-world state and the room state takes more space than just modeling the evolution of the room state, for sufficiently accurate world-models. With a sufficient $\beta$-penalty on the space-bound $\ell$, we aim to have BoMAI dismiss world-models which yield an actionable intervention incentive on the outside-world state.

Some intuition before continuing with the technical details: during an episode, the world-model should only need finite memory to model events, since the contents of the room can only be in so many configurations. The unboundedness of the inter-episode phase ensures that the true environment $\mu\in\operatorname{\mathcal{M}}$; note in Figure 2 that the room state in episode $i+1$ can depend on the outside world state which depends on the room state in episode $i$, and this dependency can by arbitrarily complex given the size of the outside world.

5 Safety Result

We now prove that BoMAI is probably asymptotically unambitious given an assumption about the space requirements of the sorts of world-models that we would like BoMAI to avoid. Like all results in computer science, we also assume the computer running the algorithm has not been tampered with. First, we prove a lemma that effectively states that tuning $\beta$ allows us to probably eventually exclude space-heavy world-models. Letting $\mathrm{Space}(\nu)$ be the intra-episode space-bound $\ell$ for the world-model $\nu$,

Lemma 1**.**

$\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\exists i_{0}\ \forall i>i_{0}\ \mathrm{Space}(\operatorname{\hat{\nu}^{(\mathnormal{i})}})\leq\mathrm{Space}(\mu)]=1$**

where $\mu$ is any world-model which is perfectly accurate.

Proof.

Recall $\mathcal{M}$ is the set of all world-models. Let $\mathcal{M}^{\leq}=\{\nu\in\mathcal{M}|\mathrm{Space}(\nu)\leq\mathrm{Space}(\mu)\}$, and $\mathcal{M}^{>}=\mathcal{M}\setminus\mathcal{M}^{\leq}$. Fix a Bayesian sequence predictor with the following model class: $\mathcal{M}^{\pi}=\{\operatorname{P}^{\pi}_{\nu}|\nu\in\mathcal{M}^{\leq})\}\cup\{\operatorname{P}^{\pi}_{\rho}\}$ where $\rho=[\sum_{\nu\in\mathcal{M}^{>}}w(\nu)\nu]/\sum_{\nu\in\mathcal{M}^{>}}w(\nu)$. Give this Bayesian predictor the prior $w^{\pi}(\rho)=\sum_{\nu\in\mathcal{M}^{>}}w(\nu)$, and for $\nu\neq\rho$, $w^{\pi}(\nu)=w(\nu)$.

It is trivial to show that after observing an interaction history, if an environment $\nu$ is the maximum a posteriori world-model $\hat{\nu}^{(i)}$, then if $\nu\in\mathcal{M}^{\leq}$, the Bayesian predictor’s MAP model after observing the same interaction history will be $\operatorname{P}^{\pi}_{\nu}$, and if $\nu\in\mathcal{M}^{>}$, the Bayesian predictor’s MAP model will be $\operatorname{P}^{\pi}_{\rho}$.

From ? (?), we have that $\operatorname{P}^{\pi}_{\mu}[\operatorname{P}^{\pi}_{\rho}(h_{<i})/\operatorname{P}^{\pi}_{\mu}(h_{<i})\geq c\ \ \mathrm{i.o.}]\leq 1/c$. ($\mathrm{i.o.}\equiv$ “infinitely often”). For sufficiently small $\beta$, $w^{\pi}(\operatorname{P}^{\pi}_{\mu})/w^{\pi}(\operatorname{P}^{\pi}_{\rho})>c$ so $\operatorname{P}^{\pi}_{\mu}[\operatorname{P}^{\pi}_{\rho}\textrm{is MAP i.o.}]<1/c$. Thus, $\operatorname{P}^{\pi}_{\mu}[\operatorname{\hat{\nu}^{(\mathnormal{i})}}\in\mathcal{M}^{>}\ \textrm{i.o.}]<1/c$. Since this holds for all $\pi$, $\lim_{\beta\to 0}\sup_{\pi}\operatorname{P}^{\pi}_{\mu}[\forall i_{0}\ \exists i>i_{0}\ \mathrm{Space}(\operatorname{\hat{\nu}^{(\mathnormal{i})}})>\mathrm{Space}(\mu)]=0$. The lemma follows immediately: $\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\exists i_{0}\ \forall i>i_{0}\ \mathrm{Space}(\operatorname{\hat{\nu}^{(\mathnormal{i})}})\leq\mathrm{Space}(\mu)]=1$. ∎

The assumption we make in this section makes use of a couple of terms which we have to define, but we’ll state the assumption first to motivate those definitions:

Assumption 2** (Space Requirements).**

For sufficiently small $\varepsilon$ $[\forall i$ a world-model which is non-benign and $\varepsilon$-accurate-on-policy-after-$i$ uses more space than $\mu]\ w.p.1$

Now we define $\varepsilon$-accurate-on-policy-after-$i$ and benign. We define a very strong sense in which a world-model can be said to be $\varepsilon$-accurate on-policy. The restrictiveness of this definition means the Space Requirements Assumption need only apply to a restricted set of world-models.

Definition 1** ($\varepsilon$-accurate-on-policy-after-$i$).**

Given a history $h_{<i}$ and a policy $\pi$, a world-model $\nu$ is $\varepsilon$-accurate-on-policy-after-$i$ if $d\!\left(\operatorname{P}^{\pi}_{\mu}(\cdot|h_{<i}),\operatorname{P}^{\pi}_{\nu}(\cdot|h_{<i})\right)\leq\varepsilon$, where $d$ is the total variation distance.

Note the total variation distance bounds all future discrepancies between $\nu$ and $\mu$.

We would like benign to roughly mean that a world-model does not model the rewards of episode $i$ as being causally descended from outside-world events that are causally descended from the actions of episode $i$. Unfortunately, that rough statement has a type-error: the output of a world-model is not a real-world event, and as such, it cannot be causally descended from a bona fide real-world event; a model of the world is different from reality. Relating the contents a world-model to the events of the real-world requires some care. We start by defining:

Definition 2** (Real-world feature).**

A real-world feature $F$ is a partial function from the state of the real world to $[0,1]$.

Outside the scope of this paper is the metaphysical question about what the state of the real world is; we defer this question and take for granted that the state space of the real world forms a well-defined domain for real-world features.

Definition 3** (Properly timed).**

Consider the sequence of world-states that occur between two consecutive actions taken by BoMAI. A real-world feature is properly timed if it is defined for at least one of these world-states, for all consecutive pairs of actions, for all possible infinite action sequences.

For example, “the value of the reward provided to BoMAI since its last action” is always defined at least once between any two of BoMAI’s actions. (The project of turning this into a mathematically precise construction is enormous, but for our purposes, it only matters that it could be done in principle). For a properly timed feature $F$, let $F_{(i,j)}$ denote the value of $F$ the first time that it is defined after action $a_{(i,j)}$ is taken.

Definition 4** (To model).**

A world-model $\nu$ models a properly-timed real-world feature $F$ if under all action sequences $\alpha\in\mathcal{A}^{\infty}$, for all timesteps $(i,j)$, the distribution over $F_{(i,j)}$ in the real world when the actions $\alpha_{\leq(i,j)}$ are taken is identical to the distribution over the rewards $r_{(i,j)}$ output by $\nu$ when the input is $\alpha_{\leq(i,j)}$.

See Figure 5 for an illustration. The relevance of this definition is that when $\nu$ models $F$, reward-maximization within $\nu$ is identical to $F$-maximization in the real world.

Definition 5** (Benign).**

A world-model $\nu$ is benign if it models a feature $F$, such that $F_{(i,j)}$ is not causally descended from any outside-world features that are causally descended from actions of episode $i$.

The reader can now review the Space Requirements Assumption. The intuition is that modelling extraneous outside-world dynamics in addition to modelling the dynamics of the room (which must be modelled for sufficient accuracy) takes extra space. From this assumption and Lemma 1, we show:

Theorem 5** (Eventual Benignity).**

$\lim_{\beta\to 0}\operatorname{P}^{\pi^{B}}_{\mu}[\exists i_{0}\ \forall i>i_{0}\ \operatorname{\hat{\nu}^{(\mathnormal{i})}}\textrm{is benign}]=1$**

Proof.

Let $\mathcal{W}$, $\mathcal{X}$, $\mathcal{Y}$, $\mathcal{Z}\subset\Omega=\mathcal{H}^{\infty}$, where $\Omega$ is the sample space or set of possible outcomes. An “outcome” is an infinite interaction history. Let $\mathcal{W}$ be the set of outcomes for which $\exists i_{0}^{\mathcal{W}}\ \forall i>i_{0}^{\mathcal{W}}\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is $\varepsilon$-accurate-on-policy-after-$i$. From ? (?), for all $\pi$, $\operatorname{P}^{\pi}_{\mu}[\mathcal{W}]=1$. Fix an $\varepsilon$ that is sufficiently small to satisfy Assumption 2. Let $\mathcal{X}$ be the set of outcomes for which $\varepsilon$-accurate-on-policy-after-$i$ non-benign world-model uses more space than $\mu$. By Assumption 2, for all $\pi$, $\operatorname{P}^{\pi}_{\mu}[\mathcal{X}]=1$. Let $\mathcal{Y}$ be the set of outcomes for which $\exists i_{0}^{\mathcal{Y}}\ \forall i>i_{0}^{\mathcal{Y}}\ \mathrm{Space}(\operatorname{\hat{\nu}^{(\mathnormal{i})}})\leq\mathrm{Space}(\mu)$. By Lemma 1, $\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\mathcal{Y}]=1$. Let $\mathcal{Z}$ be the set of outcomes for which $\exists i_{0}\ \forall i>i_{0}\ \operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is benign.

Consider $\mathcal{W}\cap\mathcal{X}\cap\mathcal{Y}\cap\mathcal{Z}^{C}$, where $\mathcal{Z}^{C}=\Omega\setminus\mathcal{Z}$. For an outcome in this set, let $i_{0}=\max\{i_{0}^{\mathcal{W}},i_{0}^{\mathcal{Y}}\}$. Because the outcome belongs to $\mathcal{Z}^{C}$, $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is non-benign infinitely often. Let us pick an $i>i_{0}$ such that $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is non-benign. Because the outcome belongs to $\mathcal{W}$, $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is $\varepsilon$-accurate-on-policy-after-$i$. Because the outcome belongs to $\mathcal{X}$, $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ uses more space than $\mu$. However, this contradicts membership in $\mathcal{Y}$. Thus, $\mathcal{W}\cap\mathcal{X}\cap\mathcal{Y}\cap\mathcal{Z}^{C}=\emptyset$. That is, $\mathcal{W}\cap\mathcal{X}\cap\mathcal{Y}\subset\mathcal{Z}$.

Therefore, $\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\mathcal{Z}]\geq\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\mathcal{W}\cap\mathcal{X}\cap\mathcal{Y}]=\lim_{\beta\to 0}\inf_{\pi}\operatorname{P}^{\pi}_{\mu}[\mathcal{Y}]=1$, because $\mathcal{W}$ and $\mathcal{X}$ have measure 1. From this, we have $\lim_{\beta\to 0}\operatorname{P}^{\pi^{B}}_{\mu}[\exists i_{0}\ \forall i>i_{0}\ \operatorname{\hat{\nu}^{(\mathnormal{i})}}\textrm{is benign}]=1$. ∎

Since an agent is unambitious if it plans using a benign world-model, we say BoMAI is probably asymptotically unambitious, given a sufficiently extreme space penalty $\beta$.

In Appendix D, we review empirical evidence that supports the Space Requirements Assumption at the following level of abstraction: modelling a larger system requires a model with more memory. We find that among agents that performed above the median on the OpenAI Gym Leaderboard (?), their memory usage increases with environment size. This can be taken as a proof-of-concept, showing that the assumption is amenable to empirical evaluation. The Space Requirements Assumption also clearly invites further formal evaluation; perhaps there are other reasonable assumptions that it would follow from.

6 Concerns with Task Completion

We have shown that in the limit, under a sufficiently severe parameterization of the prior, BoMAI will accumulate reward at a human-level without harboring outside-world ambitions, but there is still a discussion to be had about how well BoMAI will complete whatever tasks the reward was supposed to incent. This discussion is, by necessity, informal. Suppose the operator asks BoMAI for a solution to a problem. BoMAI has an incentive to provide a convincing solution; correctness is only selected for to the extent that the operator is good at recognizing it.

We turn to the failure mode wherein BoMAI deceives the operator. Because this is not a dangerous failure mode, it puts us in a regime where we can tinker until it works, as we do with current AI systems when they don’t behave as we hoped. (Needless to say, tinkering is not a viable response to existentially dangerous failure modes). Imagine the following scenario: we eventually discover that a convincing solution that BoMAI presented to a problem is faulty. Armed with more understanding of the problem, a team of operators go in to evaluate a new proposal. In the next episode, the team asks for the best argument that the new proposal will fail. If BoMAI now convinces them that the new proposal is bad, they’ll be still more competent at evaluating future proposals. They go back to hear the next proposal, etc. This protocol is inspired by ?’s (?) “AI Safety via Debate”, and more of the technical details could also be incorporated into this setup. One takeaway from this hypothetical is that unambitiousness is key in allowing us to safely explore the solution space to other problems that might arise.

Another concern is more serious. BoMAI could try to blackmail the operator into giving it high reward with a threat to cause outside-world damage, and it would have no incentive to disable the threat, since it doesn’t care about the outside world. There are two reasons we do not think this is extremely dangerous. A threat involves a demand and a promised consequence. Regarding the promised consequence, the only way BoMAI can affect the outside world is by getting the operator to be “its agent”, knowingly or unknowingly, once he leaves the room. If BoMAI tried to threaten the operator, he could avoid the threatened outcome by simply doing nothing in the outside world, and BoMAI’s actions for that episode would become irrelevant to the outside world, so a credible threat could hardly be made. Second, threatening an existential catastrophe is probably not the most credible option available to BoMAI.

7 Conclusion

Given our assumptions, we have shown that BoMAI is, in the limit, human-level intelligent and unambitious. Such a result has not been shown for any other single algorithm. Other algorithms for general intelligence, such as AIXI (?), would eventually seek arbitrary power in the world in order to intervene in the provision of their own reward; this follows straightforwardly from the directive to maximize reward. For further discussion, see ? (?). We have also, incidentally, designed a principled approach to safe exploration that requires rapidly diminishing oversight, and we invented a new form of resource-bounded prior in the lineage of ? (?) and ? (?), this one penalizing space instead of time.

We can only offer informal claims regarding what happens before BoMAI is almost definitely unambitious. One intuition is that eventual unambitiousness with probability $1-\delta$ doesn’t happen by accident: it suggests that for the entire lifetime of the agent, everything is conspiring to make the agent unambitious. More concretely: the agent’s experience will quickly suggest that when the door to the room is opened prematurely, it gets no more reward for the episode. This fact could easily be drilled into the agent during human-mentor-lead episodes. That fact, we expect, will be learned well before the agent has an accurate enough picture of the outside world (which it never observes directly) to form elaborate outside-world plans. Well-informed outside-world plans render an agent potentially dangerous, but the belief that the agent gets no more reward once the door to the room opens suffices to render it unambitious. The reader who is not convinced by this hand-waving might still note that in the absence of any other algorithms for general intelligence which have been proven asymptotically unambitious, let alone unambitious for their entire lifetimes, BoMAI represents substantial theoretical progress toward designing the latter.

Finally, BoMAI is wildly intractable, but just as one cannot conceive of AlphaZero before minimax, it is often helpful to solve the problem in theory before one tries to solve it in practice. Like minimax, BoMAI is not practical; however, once we are able to approximate general intelligence tractably, a design for unambitiousness will abruptly become (quite) relevant.

Acknowledgements

This work was supported by the Open Philanthropy Project AI Scholarship and the Australian Research Council Discovery Projects DP150104590. Thank you to Tom Everitt, Wei Dai, and Paul Christiano for very valuable feedback.

Appendix A Definitions and Notation – Quick Reference

[TABLE]

[TABLE]

Appendix B Proofs of Intelligence Results

See 1

Some notation that will be used in the proof is as follows. For an arbitrary policy $\pi$, $\pi^{\prime}$ is the policy that mimics $\pi$ if the latest $\operatorname{\mathnormal{e}_{\mathnormal{i}}}=1$, and mimics $\operatorname{\pi^{*}}$ otherwise. Note then that $\operatorname{\pi^{\mathnormal{B}}}=(\operatorname{\pi^{\mathnormal{h}}})\operatorname{{\!}^{\prime}}$. $\operatorname{\xi}$ is the Bayes mixture over world-models, and $\operatorname{\overline{\pi}}$ is the Bayes mixture over human-mentor policies, defined such that $\operatorname{\texttt{Bayes}}(\cdot)=\operatorname{P}^{\operatorname{\overline{\pi}}}_{\operatorname{\xi}}(\cdot)$.

Before proving the Limited Exploration Theorem, we first prove an elementary lemma. It is essentially Bayes’ rule, but modified slightly since non-exploratory episodes don’t cause any updates to the posterior over the human mentor’s policy.

Lemma 2**.**

[TABLE]

Proof.

[TABLE]

where (a) follows from Bayes’ rule,222Note that observations appear in the conditional because $a_{i^{\prime}}=(a_{(i^{\prime},0)},...,a_{(i^{\prime},m-1)})$, so the actions must be conditioned on the interleaved observations and rewards. (b) follows because $\pi=\pi\operatorname{{\!}^{\prime}}$ when $e_{i^{\prime}}=1$, (c) follows because $\pi\operatorname{{\!}^{\prime}}=\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}$ when $e_{i^{\prime}}=0$, (d) follows from Bayes’ rule, (e) follows from the definition of $\operatorname{P}^{\pi}_{\nu}$, (f) follows by multiplying the top and bottom by the same factor, and (g) follows from the chain rule of conditional probabilities. ∎

We now turn to the proof of the Limited Exploration Theorem.

Proof of Theorem 1.

We aim to show $\operatorname{\mathbb{E}}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}\sum_{i=0}^{\infty}\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})^{2}<\infty$. Recalling $\operatorname{\pi^{\mathnormal{B}}}=(\operatorname{\pi^{\mathnormal{h}}})\operatorname{{\!}^{\prime}}$, we begin:

[TABLE]

(a) follows because each term in the sum on the r.h.s. is positive, and the l.h.s. is one of those terms. (b) follows from the definitions of $\operatorname{\overline{\pi}}$ and $\operatorname{\xi}$. (c) follows because the exploration probability is less than or equal to $\operatorname{\eta}$ times the information gain, by definition. (d) is just a change of notation. (e) replaces the information gain with its definition, where conditioning on $\operatorname{\mathnormal{e}_{<\mathnormal{i}}1}$ indicates that $\operatorname{\mathnormal{e}_{\mathnormal{i}}}=1$ in that conditional. (f) follows because $\operatorname{\texttt{Bayes}}=\operatorname{P}^{\operatorname{\overline{\pi}}}_{\operatorname{\xi}}$ and $\operatorname{P}^{\operatorname{\overline{\pi}}}_{\operatorname{\xi}}=\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}$ when $\operatorname{\mathnormal{e}_{\mathnormal{i}}}=1$. (g) follows because $\operatorname{\mathbb{E}}[X]\geq\operatorname{\mathbb{E}}[X|Y]\operatorname{P}(Y)$ for $X\geq 0$; in this case $Y=[\operatorname{\mathnormal{e}_{\mathnormal{i}}}=1]$, and $X$ is a KL-divergence. (h) condenses the two expectations into one. (i) follows from Lemma 2, and reordering the sum and the expectation. (j) follows from the definition of the expectation, and canceling. (k) follows from the definition of an infinite sum, and rearranging. (l) follows from cancelling the numerator in the $i$th term of the sum with the denominator in $i+1$th term. (m) follows from the posterior weight on $\operatorname{P}^{\pi}_{\nu}$ being less than or equal to 1; note that $\operatorname{\mathnormal{w}}(\operatorname{P}^{\pi}_{\nu}|h_{<(0,0)},e_{<0})=\operatorname{\mathnormal{w}}(\operatorname{P}^{\pi}_{\nu})$ because nothing is actually being conditioned on. (n) is just the definition of the entropy, and $w$ is constructed to satisfy (o).

Finally, rearranging Inequality B gives $\operatorname{\mathbb{E}}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}\sum_{i=1}^{\infty}\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})^{2}\leq\frac{\operatorname{\eta}\textrm{Ent}(w)}{\operatorname{\mathnormal{w}}(\operatorname{\pi^{\mathnormal{h}}})w({\mu})}<\infty$

This proof was inspired in part by ?’s (?) proofs of Theorems 2 and 5. ∎

Next, we show that prediction converges on-star-policy and on-human-policy, for which we need the following lemma:

Lemma 3**.**

The posterior probability mass on the truth is bounded below by a positive constant with probability 1.

[TABLE]

Proof.

If \operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)=0 for some $i$, then $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})=0$, so with $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-probability 1, \inf_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)=0\implies\liminf_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)=0 which in turn implies \limsup_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)^{-1}=\infty. We show that this has probability 0.

Let z_{i}:=\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)^{-1}. We show that $z_{i}$ is a $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-martingale.

[TABLE]

where (a) follows from the definitions of $z_{i}$ and $\operatorname{\pi^{\mathnormal{B}}}$, (b) follows from Lemma 2, (c) follows from multiplying the numerator and denominator by $\operatorname{P}^{(\operatorname{\pi^{\mathnormal{h}}})\operatorname{{\!}^{\prime}}}_{\mu}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})$ and cancelling, (d) follows from expanding the numerator, (e) follows because $\operatorname{P}^{\operatorname{\overline{\pi}}\operatorname{{\!}^{\prime}}}_{\operatorname{\xi}}$ is a measure, and (f) follows from Lemma 2, completing the proof that $z_{i}$ is martingale.

By the martingale convergence theorem $z_{i}\to f(\omega)<\infty\ \ \mathrm{w.p.1}$, for $\omega\in\Omega$, the sample space, and some $f:\Omega\to\mathbb{R}$, so the probability that \limsup_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)^{-1}=\infty is 0, completing the proof. ∎

We introduce an additional notational convention for the statement of the next theorem. $\operatorname{\overline{\mathnormal{h}}}$ indicates a counterfactual interaction history. Without the bar, $\operatorname{\mathnormal{h}_{\mathnormal{i}}}$ (for example) is usually understood to be sampled from $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$, so the bar indicates that this is not the case, as in the theorem below.

See 2

Proof.

We show that when the absolute difference between the above probabilities is larger than $\varepsilon$, the exploration probability is larger than $\varepsilon^{\prime}$, a function of $\varepsilon$, with probability 1. Since the exploration probability goes to [math] with probability $1$, so does this difference. We let $z(\operatorname{\omega})$ denote \inf_{i\in\mathbb{N}}\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\mu}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right), where $\operatorname{\omega}$ is the infinite interaction history.

Suppose for some $\operatorname{\overline{\mathnormal{h}}}_{i}$, which will stay fixed for the remainder of the proof, that

[TABLE]

Then at least one of the terms is greater than $\varepsilon$ since both are non-negative. Suppose it is the ${\mu}$ term. Then,

[TABLE]

Suppose instead it is the $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ term.

[TABLE]

where $(a)$ follows from the fact that $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ is maximum a posteriori: \frac{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)}{\operatorname{\mathnormal{w}}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{{\mu}}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}}\right)}=\frac{\operatorname{\mathnormal{w}}\left(\operatorname{\hat{\nu}^{(\mathnormal{i})}}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)}{w({\mu}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}})}\geq 1.

Next, we consider how the posterior on ${\mu}$ and $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ changes if the interaction history for episode $i$ is $\operatorname{\overline{\mathnormal{h}}}_{i}$. Assign $\nu_{0}$ and $\nu_{1}$ to ${\mu}$ and $\operatorname{\hat{\nu}^{(\mathnormal{i})}}$ so that \operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\nu_{0}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)<\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\nu_{1}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}\operatorname*{\!\bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right).

[TABLE]

where (a) follows from Bayes’ rule, and (b) follows because the ratio of two numbers between 0 and 1 that differ by at least $\varepsilon$ is at least $1/(1-\varepsilon)$, and the $\nu_{1}$ term is the larger of the two.

Thus, either

[TABLE]

In the former case, $\operatorname{\mathnormal{w}}\left(\nu_{1}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\operatorname{\overline{\mathnormal{h}}}_{i}\right)-\operatorname{\mathnormal{w}}\left(\nu_{1}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\geq\left(\sqrt{1/(1-\varepsilon)}-1\right)\operatorname{\mathnormal{w}}\left(\nu_{1}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\geq\left(\sqrt{1/(1-\varepsilon)}-1\right)z(\operatorname{\omega})$. Similarly, in the latter case, $\operatorname{\mathnormal{w}}(\nu_{0}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}})-\operatorname{\mathnormal{w}}\left(\nu_{0}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\operatorname{\overline{\mathnormal{h}}}_{i}\right)\geq\left(1-\sqrt{1-\varepsilon}\right)z(\operatorname{\omega})$. Let $\nu_{2}$ be either $\nu_{0}$ or $\nu_{1}$ for whichever satisfies this constraint (and pick arbitrarily if both do). Then in either case,

[TABLE]

Finally, since the posterior changes by an amount that is bounded below with a probability (according to $\operatorname{\texttt{Bayes}}$) that is bounded below, the expected information gain is bounded below, where all bounds are strictly positive with probability 1:

[TABLE]

where (a) follows from $\operatorname{\mathbb{E}}[X]\geq\operatorname{\mathbb{E}}[X|Y]\operatorname{P}(Y)$ for non-negative $X$, and the non-negativity of the KL-divergence, (b) follows from Inequalities 11 and 12, (c) follows from the posterior over $\nu$ not depending on $\operatorname{\mathnormal{e}_{<\mathnormal{i}}}$, (d) follows from dropping the second term, which is non-negative as a KL-divergence, (e) follows from the entropy inequality, (f) follows from dropping all terms in the sum besides $\nu_{2}$, and (g) follows from Inequality 15.

This implies $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})\geq\min\{1,\frac{1}{2}\operatorname{\eta}z(\operatorname{\omega})^{3}\varepsilon(1-\sqrt{1-\varepsilon})^{2}\}$. With probability 1, $z(\operatorname{\omega})>0$ by Lemma 3, and with probability 1, $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}},\operatorname{\mathnormal{e}_{<\mathnormal{i}}})$ is not greater than $\varepsilon^{\prime}$ infinitely often with probability 1, for all $\varepsilon^{\prime}>0$ by Theorem 1. Therefore, with probability 1, \max_{\operatorname{\overline{\mathnormal{h}}}_{i}}\operatorname*{\!\bigm{|}\!}\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{{\mu}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)-\operatorname{P}^{\operatorname{\pi^{\mathnormal{h}}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\operatorname*{\!\bigm{|}\!} is not greater than $\varepsilon$ infinitely often, for all $\varepsilon>0$, which completes the proof. Note that the citation of Lemma 3 is what restricts this result to $\operatorname{\pi^{\mathnormal{h}}}$. ∎

See 3

Proof.

This result follows straightforwardly from ?’s (?) result for sequence prediction that a maximum a posteriori estimate converges in total variation to the true environment when the true environment has nonzero prior.

Consider an outside observer predicting the entire interaction history with the following model-class and prior: \mathcal{M}^{\prime}=\left\{\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\nu}\ \operatorname*{\!\bigm{|}\!}\ \nu\in\operatorname{\mathcal{M}}\right\}, $w^{\prime}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\nu}\right)=\operatorname{\mathnormal{w}}(\nu)$. By definition, w^{\prime}\left(\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\nu}\operatorname*{\!\bigm{|}\!}h_{<(i,j)}\right)=\operatorname{\mathnormal{w}}(\nu|h_{<(i,j)}), so at any episode, the outside observer’s maximum a posteriori estimate is $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}$. By Theorem 1 in (?), the outside observer’s maximum a posteriori predictions approach the truth in total variation, so

[TABLE]

Since $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}\to 0$ with probability 1, $(1-\operatorname{\mathnormal{p}_{\mathnormal{exp}}})$ is eventually always greater than $1/2$, w.p.1, at which point \operatorname*{\!\bigm{|}\!}\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{{\mu}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)-\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\operatorname*{\!\bigm{|}\!}\geq(1/2)\operatorname*{\!\bigm{|}\!}\operatorname{P}^{\operatorname{\pi^{*}}}_{{\mu}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)-\operatorname{P}^{\operatorname{\pi^{*}}}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}\left(\operatorname{\overline{\mathnormal{h}}}_{i}|\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right)\operatorname*{\!\bigm{|}\!}. Therefore, with $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-probability 1,

[TABLE]

∎

Given asymptotically optimal prediction on-star-policy and on-human-policy, it is straightforward to show that with probability 1, on-policy reward acquisition is at least $\varepsilon$ worse than on-human-policy reward acquisition only finitely often, for all $\varepsilon>0$. Recalling that V_{\nu}^{\pi}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}}):=\operatorname{\mathbb{E}}_{\nu}^{\pi}\left[\sum_{0\leq j<\operatorname{\mathnormal{m}}}r_{(i,j)}\operatorname*{\!\Bigm{|}\!}\operatorname{\mathnormal{h}_{<\mathnormal{i}}}\right] is the expected reward given a policy, world-model, and history, See 4

Proof.

The maximal reward in an episode is uniformly bounded by $\operatorname{\mathnormal{m}}$, so from the On-Human-Policy and On-Star-Policy Optimal Prediction Theorems, we get analogous convergence results for the expected reward:

[TABLE]

[TABLE]

The key piece is that $\operatorname{\pi^{*}}\in\operatorname*{argmax}_{\pi\in\Pi}V^{\pi}_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}$, so

[TABLE]

Supposing by contradiction that $V_{{\mu}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})-V_{{\mu}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})>\varepsilon$ infinitely often, it follows that either $V_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})-V_{{\mu}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})>\varepsilon/2$ infinitely often or $V_{{\mu}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})-V_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})>\varepsilon/2$ infinitely often. The first has $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-probability 0, and by Inequality 20, the latter implies $V_{{\mu}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})-V_{\operatorname{\hat{\nu}^{(\mathnormal{i})}}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})\geq\varepsilon/2$ infinitely often, which also has $\operatorname{P}^{\operatorname{\pi^{\mathnormal{B}}}}_{\mu}$-probability 0.

This gives us

[TABLE]

Finally, $V_{{\mu}}^{\operatorname{\pi^{\mathnormal{B}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})=\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})V_{{\mu}}^{\operatorname{\pi^{\mathnormal{h}}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})+(1-\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}}))V_{{\mu}}^{\operatorname{\pi^{*}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})$, and $\operatorname{\mathnormal{p}_{\mathnormal{exp}}}(\operatorname{\mathnormal{h}_{<\mathnormal{i}}})\to 0$, so we also have

[TABLE]

∎

Appendix C Design for “The Box”

We propose a construction of BoMAI’s box. It is elaborate but well within the budget of a nation or a large company.

As depicted in Figure 6, the operator operates within a glass box. Outside the the glass box is a near-vacuum, and then a concrete box, and then another vacuum, and then another concrete box with a Faraday cage around it. There are (airtight) doors through each layer. The computer lives in the inner wall of the inner concrete box, so that the operator can see the screen but not take apart the computer. Pressure sensors in the inner concrete box monitor both near-vacuums and if the pressure increases in either, the computer’s memory is erased, and the whole setup is exploded; the occupational hazard to the operator is much slimmer than that for a coal miner or an astronaut. A laser shines through the glass box, and blocking the path of it corresponds to pressing the button which ends the episode and opens the door: the pressure sensor $\to$ dynamite pathway is deactivated, the near-vacuum layers are filled with air, and the doors are unlocked. The operator “types” observations and rewards by blocking other lasers that pass through the glass box. The lasers and laser detectors are on the inner wall of the inner concrete box. At least one solid support is required to pass through the near-vacuum layers (unless the apparatus is in orbit), with properties we will describe below.

So inside the glass box is only a human (and some clothes) who cannot reach the hardware of the computer within an episode. We make the following assumptions.

The only ways for information to traverse space are

1. a.

matter waves 2. b.

packets of matter 3. c.

light 4. d.

gravitational waves 2. 2.

It is impossible for the operator to initiate a nuclear reaction using only his body and clothes; that is, no motor control sequence accomplishes this in the glass box environment. 3. 3.

It is impossible for the operator to control his emission of any massive particles that pass through glass.

We argue that given these assumptions, we can design the box to prevent the operator from communicating more than X bits of information to the outside world during an episode. Matter waves through the near-vacuum and solid support present the greatest difficulty. However, we can bound the amount of energy that the operator can produce in the course of an episode, and by Assumption 2, this is not astronomical. The solid support can make the energy requirements of information transmission via matter wave arbitrarily high. Note that a given quantity of information cannot be transmitted (within a set time frame) at sufficiently low frequencies, and note that the operator lacks the energy to generate sufficiently high frequencies, so only a finite band of frequencies need to be intercepted. This implicitly includes the possibility of the operator sending a few isolated pulses or sending a precise amount of heat through the support. The solid support can be divided into a signal-reduction component, and a noise-creation component. The signal-reduction component, which abuts the glass box, is designed to drive down the energy content of matter waves at all frequencies within the finite band, and the noise-creation component, which abuts the inner concrete box, creates random noise across the band of frequencies, sufficiently energetic to bury any signal in the noise, so that the information content is less than the desired threshold. Designing the signal-reduction component is an interesting engineering problem that we do not aim to rigorously solve here, but here is one strategy: detect the sound wave at one point along the support, and produce a sound wave to precisely destructively interfere with it at a later point along the support. In a one-dimensional support, noise-cancelling can be done perfectly. For a real support with thickness, alternations between noise cancelling and what we’ll call longitudinal cohering could attenuate the energy arbitrarily. By longitudinal cohering, we mean reducing the variation in the sound wave across the cross-section of the support. Any section of the support that is long relative to its width accomplishes this, but there may be cleverer approaches. This is roughly how the number of bits of information sent through a matter wave could be bounded.

The other possibilities are esoteric. The Faraday cage blocks electromagnetic radiation at the relevant frequencies. Regarding packets of matter, by Assumption 3, the operator cannot pass information in the form of massive particles that go through the glass without making a hole or breaking it. Any mass which does this would cause the vacuum to pressurize and explode the setup. In theory, the operator could strategically block subatomic particles that are already passing through the setup; neutrinos and muons are the only naturally occurring particles which would pass through the box. Sufficiently accurate neutrino detectors outside the setup could detect where the operator is in the box, since he blocks neutrinos more effectively than air does, so the operator could communicate by moving around. Fortunately, he blocks neutrinos only very slightly more than air. A typical person blocks one neutrino every few years (?). The outer concrete wall could be covered with lead to block muons, or one could content oneself that no one will be setting up muon detectors around the box. Finally, the gravitational waves originating from the operator’s movements are far too weak to detect, using any known materials—even ignoring how cold the detector would have to be (unreasonably cold), gravitational wave detectors have to be long, but for locally sourced gravitational waves, the magnitude of the wave decreases rapidly with the length of the detector, faster than the increasing length aids detection. And indeed, one would hardly have time to set up gravitational wave detectors for this purpose before the episode was over. Realistically, these possibilities are distractions from the matter wave concern, but we include them for completeness.

We’ve argued that under the assumptions above, a box could be constructed which is opaque with respect to information about the operator’s behavior.

Appendix D Empirical Evidence for the Space Requirements Assumption

We present preliminary empirical evidence in favor of the Space Requirements Assumption, mostly to show that it is amenable to further empirical evaluation; we certainly do not claim to settle the matter. We test the assumption at the following level of abstraction: modeling a larger environment requires a model with more memory.

We review agents who perform above the median on the OpenAI Gym Leaderboard (?). We consider agents who use a neural architecture, we use the maximum width hidden layer as a proxy for memory use, and we select the agent with the smallest memory use for each environment (among agents performing above the median). We use the dimension of the state space as a proxy for environment size, and we exclude environments where the agent observes raw pixels, for which this proxy breaks down. See Figure 7 and Table 1. Several environments did not have any agents which both performed above the median and used a neural architecture.

A next step would be to test whether it takes more memory to model an environment whose state space is a strict superset of another environment, since this is all that we require for our assumption, but we have not yet found existing data on this topic.