To Know is to Construct: Schema-Constrained Generation for Agent Memory

TL;DR

This paper introduces SCG-MEM, a schema-constrained generative memory system for LLMs that improves long-term memory recall by enforcing schema-based constraints, reducing hallucinations, and enabling multi-hop reasoning.

Contribution

It proposes a novel schema-constrained generative approach for agent memory, integrating dynamic schemas and associative graphs to enhance recall accuracy and reasoning capabilities.

Findings

SCG-MEM outperforms retrieval-based baselines on the LoCoMo benchmark.

The schema constraint reduces structural hallucinations during memory access.

Multi-hop reasoning is facilitated through an associative graph structure.

Abstract

Constructivist epistemology argues that knowledge is actively constructed rather than passively copied. Despite the generative nature of Large Language Models (LLMs), most existing agent memory systems are still based on dense retrieval. However, dense retrieval heavily relies on semantic overlap or entity matching within sentences. Consequently, embeddings often fail to distinguish instances that are semantically similar but contextually distinct, introducing substantial noise by retrieving context-mismatched entries. Conversely, directly employing open-ended generation for memory access risks "Structural Hallucination" where the model generates memory keys that do not exist in the memory, leading to lookup failures. Inspired by this epistemology, we posit that memory is fundamentally organized by cognitive schemas, and valid recall must be a generative process performed within these schematic structures. To realize this, we propose SCG-MEM, a schema-constrained generative memory architecture. SCG-MEM reformulates memory access as Schema-Constrained Generation. By maintaining a dynamic Cognitive Schema, we strictly constrain LLM decoding to generate only valid memory entry keys, providing a formal guarantee against structural hallucinations. To support long-term adaptation, we model memory updates via assimilation (grounding inputs into existing schemas) and accommodation (expanding schemas with novel concepts). Furthermore, we construct an Associative Graph to enable multi-hop reasoning through activation propagation. Experiments on the LoCoMo benchmark show that SCG-MEM substantially improves performance across all categories over retrieval-based baselines.