STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory

TL;DR

STaR introduces a scalable, task-conditioned multimodal memory framework for long-horizon robot reasoning, enabling efficient retrieval and reasoning over diverse, dynamic environments for navigation and task execution.

Contribution

The paper presents STaR, a novel framework that constructs a generalizable, multimodal long-term memory and a scalable retrieval algorithm based on the Information Bottleneck principle for robotic applications.

Findings

Outperforms baselines on NaVQA and WH-VQA benchmarks

Achieves higher success rates and lower spatial errors

Demonstrates robustness and scalability on real robots

Abstract

Mobile robots are often deployed over long durations in diverse open, dynamic scenes, including indoor setting such as warehouses and manufacturing facilities, and outdoor settings such as agricultural and roadway operations. A core challenge is to build a scalable long-horizon memory that supports an agentic workflow for planning, retrieval, and reasoning over open-ended instructions at variable granularity, while producing precise, actionable answers for navigation. We present STaR, an agentic reasoning framework that (i) constructs a task-agnostic, multimodal long-term memory that generalizes to unseen queries while preserving fine-grained environmental semantics (object attributes, spatial relations, and dynamic events), and (ii) introduces a Scalable Task Conditioned Retrieval algorithm based on the Information Bottleneck principle to extract from long-term memory a compact, non-redundant, information-rich set of candidate memories for contextual reasoning. We evaluate STaR on NaVQA (mixed indoor/outdoor campus scenes) and WH-VQA, a customized warehouse benchmark with many visually similar objects built with Isaac Sim, emphasizing contextual reasoning. Across the two datasets, STaR consistently outperforms strong baselines, achieving higher success rates and markedly lower spatial error. We further deploy STaR on a real Husky wheeled robot in both indoor and outdoor environments, demonstrating robust long horizon reasoning, scalability, and practical utility. Project Website: https://trailab.github.io/STaR-website/