SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

TL;DR

SmartSearch demonstrates that a fully deterministic, unstructured retrieval pipeline with minimal learned components can outperform structured memory systems in conversational recall, achieving high accuracy with fewer tokens.

Contribution

This work introduces a simple, deterministic retrieval pipeline that forgoes complex structuring and learned policies, yet surpasses existing memory systems in conversational memory recall.

Findings

Achieves 98.6% recall with oracle analysis but only 22.5% gold evidence retention without ranking.

Surpasses all known memory systems on LoCoMo and LongMemEval-S benchmarks.

Uses 8.5x fewer tokens than full-context baselines.

Abstract

Recent conversational memory systems invest heavily in LLM-based structuring at ingestion time and learned retrieval policies at query time. We show that neither is necessary. SmartSearch retrieves from raw, unstructured conversation history using a fully deterministic pipeline: NER-weighted substring matching for recall, rule-based entity discovery for multi-hop expansion, and a CrossEncoder+ColBERT rank fusion stage -- the only learned component -- running on CPU in ~650ms. Oracle analysis on two benchmarks identifies a compilation bottleneck: retrieval recall reaches 98.6%, but without intelligent ranking only 22.5% of gold evidence survives truncation to the token budget. With score-adaptive truncation and no per-dataset tuning, SmartSearch achieves 93.5% on LoCoMo and 88.4% on LongMemEval-S, exceeding all known memory systems under the same evaluation protocol on both benchmarks while using 8.5x fewer tokens than full-context baselines.