Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search

TL;DR

This paper introduces SCOUT, a scene graph-based method for open-world object search that combines relational heuristics with learned utility scores, achieving efficient and effective semantic reasoning in real-world environments.

Contribution

The paper presents SCOUT, a novel scene graph-based exploration method with a procedural distillation framework, and introduces SymSearch, a symbolic benchmark for semantic reasoning evaluation.

Findings

SCOUT outperforms embedding similarity methods in accuracy.

SCOUT matches LLM performance with higher efficiency.

Successful transfer to real-world environments.

Abstract

Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on vision-language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-robot inference. Furthermore, we present SymSearch, a scalable symbolic benchmark for evaluating semantic reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and navigation constraints.