Artifacts as Memory Beyond the Agent Boundary
John D. Martin, Fraser Mince, Esra'a Saleh, Amy Pajak

TL;DR
This paper formalizes how environmental resources, termed artifacts, can serve as external memory in reinforcement learning, reducing internal memory needs and enabling more efficient policy learning.
Contribution
It introduces a mathematical framework for environmental memory in RL and demonstrates its effectiveness through experiments with spatial path observations.
Findings
Artifacts can reduce the memory required for RL agents.
Environmental observations implicitly serve as external memory.
Memory reduction occurs naturally through sensory streams.
Abstract
The situated view of cognition holds that intelligent behavior depends not only on internal memory, but on an agent's active use of environmental resources. Here, we begin formalizing this intuition within Reinforcement Learning (RL). We introduce a mathematical framing for how the environment can functionally serve as an agent's memory, and prove that certain observations, which we call artifacts, can reduce the information needed to represent history. We corroborate our theory with experiments showing that when agents observe spatial paths, the amount of memory required to learn a performant policy is reduced. Interestingly, this effect arises unintentionally, and implicitly through the agent's sensory stream. We discuss the implications of our findings, and show they satisfy qualitative properties previously used to ground accounts of external memory. Moving forward, we anticipate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
