FlashMem: Distilling Intrinsic Latent Memory via Computation Reuse

TL;DR

FlashMem introduces a novel memory distillation method for large language models, enabling efficient long-term context preservation by reusing computation and reducing inference latency.

Contribution

The paper presents FlashMem, a framework that distills intrinsic memory from reasoning states, eliminating architectural segregation and improving efficiency in LLMs.

Findings

Matches baseline performance while reducing inference latency by 5x.

Uses attention entropy to adaptively trigger memory consolidation.

Effectively bridges efficiency and persistent cognition in LLMs.

Abstract

The stateless architecture of Large Language Models inherently lacks the mechanism to preserve dynamic context, compelling agents to redundantly reprocess history to maintain long-horizon autonomy. While latent memory offers a solution, current approaches are hindered by architectural segregation, relying on auxiliary encoders that decouple memory from the reasoning backbone. We propose FlashMem, a framework that distills intrinsic memory directly from transient reasoning states via computation reuse. Leveraging the property that internal representations uniquely encode input trajectories, FlashMem identifies the last hidden state as a sufficient statistic for the interaction history. This enables a Shared-KV Consolidator to synthesize memory by attending directly to the backbone's frozen cache, eliminating redundant re-parameterization. Furthermore, a parameter-free Cognitive Monitor leverages attention entropy to adaptively trigger consolidation only when high epistemic uncertainty is detected. Experiments demonstrate that FlashMem matches the performance of heavy baselines while reducing inference latency by 5 times, effectively bridging the gap between efficiency and persistent cognition.