RT-Cache: Training-Free Retrieval for Real-Time Manipulation

TL;DR

RT-Cache is a training-free retrieval system that enables real-time robot control by caching diverse trajectories and efficiently retrieving them at test time, outperforming existing methods in success rate and speed.

Contribution

It introduces RT-Cache, a novel retrieval-as-control pipeline that replaces per-step inference with cached trajectory snippets, enabling real-time manipulation without training.

Findings

Higher success rates compared to baselines (~2x)

Approximately 30% faster in completion time

Effective adaptation to complex, contact-rich tasks without fine-tuning

Abstract

Real robots are expected to repeat the same behavior in new environments with very little new data, yet modern controllers either incur heavy per-step inference or require deployment-time fine-tuning. We propose RT-Cache, a training-free retrieval-as-control pipeline that caches diverse image action trajectories in a unified vector memory and, at test time, embeds the current frame to retrieve and replay multi-step snippets, replacing per-step model calls. A hierarchical search keeps lookups sub-second at million scale, shifting cost from compute to storage and enabling real-time control on modest GPUs. Across real-robot tasks and large open logs, RT-Cache achieves higher success and lower completion time than strong retrieval baselines (approximately x2 higher success and ~30% faster in our settings), and a single-episode anchoring study shows immediate adaptation to a more complex, contact-rich task without fine-tuning. RT-Cache turns experience into an append-only memory, offering a simple, scalable path to few-shot deployment today and a foundation for multimodal keys and optional integration with high-level policies. Project page: https://rt-cache.github.io/.