Prometheus Mind: Retrofitting Memory to Frozen Language Models

TL;DR

Prometheus Mind introduces a method to retrofit memory into frozen language models using modular adapters, enabling improved retrieval and understanding without altering the original model weights.

Contribution

This work develops a novel approach to add reversible memory to frozen language models through modular adapters and addresses key challenges in extraction, training, injection, and state collapse.

Findings

Achieves 94.4% retrieval accuracy on clean inputs

Degrades to 19.4% accuracy on informal inputs

Identifies relation classification as the main bottleneck

Abstract

Adding memory to pretrained language models typically requires architectural changes or weight modification. We present Prometheus Mind, which retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead) -- fully reversible by removing the adapters. Building this system required solving four problems: (1) Extraction -- we develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data. (2) Training -- end-to-end optimization collapses; stage-wise training of each adapter on simple proxy tasks succeeds. (3) Injection -- learned encoders fail to generalize; we find that lm_head-weight rows already provide the mapping we need, requiring no training. (4) Hidden state collapse -- transformers make ``wife'' and ``brother'' 0.98+ similar; we train projections to recover distinction (0.98 $\rightarrow$ 0.09). On PrometheusExtract-132 (132 cases), the system achieves 94.4% retrieval on clean inputs (n=54, 95% CI: [84.9%, 98.1%]), degrading to 19.4% on informal inputs with ellipsis, filler words, or implicit subjects (n=36). The primary bottleneck is relation classification (47.3% accuracy), responsible for most extraction errors.