Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agents

TL;DR

This paper introduces a cost-sensitive store routing method for memory-augmented agents, improving retrieval efficiency and accuracy by selectively accessing relevant memory stores rather than all stores.

Contribution

It formulates memory retrieval as a store-routing problem, demonstrating that learned routing mechanisms enhance scalability and performance in multi-store systems.

Findings

Oracle routing outperforms uniform retrieval in accuracy and token efficiency.

Selective retrieval reduces context tokens used without sacrificing performance.

Routing decisions are essential for scalable memory-augmented agent design.

Abstract

Memory-augmented agents maintain multiple specialized stores, yet most systems retrieve from all stores for every query, increasing cost and introducing irrelevant context. We formulate memory retrieval as a store-routing problem and evaluate it using coverage, exact match, and token efficiency metrics. On downstream question answering, an oracle router achieves higher accuracy while using substantially fewer context tokens compared to uniform retrieval, demonstrating that selective retrieval improves both efficiency and performance. Our results show that routing decisions are a first-class component of memory-augmented agent design and motivate learned routing mechanisms for scalable multi-store systems. We additionally formalize store selection as a cost-sensitive decision problem that trades answer accuracy against retrieval cost, providing a principled interpretation of routing policies.