Schrodinger's Memory: Large Language Models

TL;DR

This paper explores the memory mechanisms of Large Language Models (LLMs) using the Universal Approximation Theorem, proposing a new perspective that likens LLM memory to Schr"odinger's memory, and compares it with human memory.

Contribution

It introduces a theoretical explanation of LLM memory based on UAT and proposes a novel method to assess their memory capabilities, framing LLM memory as observable only upon querying.

Findings

LLMs exhibit memory that can be tested through specific queries.

The proposed assessment method effectively measures LLM memory capabilities.

LLM memory operates similarly to Schr"odinger's memory, being indeterminate until observed.

Abstract

Memory is the foundation of all human activities; without memory, it would be nearly impossible for people to perform any task in daily life. With the development of Large Language Models (LLMs), their language capabilities are becoming increasingly comparable to those of humans. But do LLMs have memory? Based on current performance, LLMs do appear to exhibit memory. So, what is the underlying mechanism of this memory? Previous research has lacked a deep exploration of LLMs' memory capabilities and the underlying theory. In this paper, we use Universal Approximation Theorem (UAT) to explain the memory mechanism in LLMs. We also conduct experiments to verify the memory capabilities of various LLMs, proposing a new method to assess their abilities based on these memory ability. We argue that LLM memory operates like Schr\"odinger's memory, meaning that it only becomes observable when a specific memory is queried. We can only determine if the model retains a memory based on its output in response to the query; otherwise, it remains indeterminate. Finally, we expand on this concept by comparing the memory capabilities of the human brain and LLMs, highlighting the similarities and differences in their operational mechanisms.