On Extending Semantic Abstraction for Efficient Search of Hidden Objects

TL;DR

This paper proposes an extension of semantic abstraction techniques to efficiently locate and complete 3D models of hidden, occluded objects in scenes, enabling household robots to find lost items faster and more accurately.

Contribution

It introduces a novel method for localizing hidden objects using relevancy maps as abstract representations, improving search efficiency over naive methods.

Findings

Accurately localizes hidden objects in 3D on the first try

Significantly faster search compared to random search

Enhances household robot capabilities for object retrieval

Abstract

Semantic Abstraction's key observation is that 2D VLMs' relevancy activations roughly correspond to their confidence of whether and where an object is in the scene. Thus, relevancy maps are treated as "abstract object" representations. We use this framework for learning 3D localization and completion for the exclusive domain of hidden objects, defined as objects that cannot be directly identified by a VLM because they are at least partially occluded. This process of localizing hidden objects is a form of unstructured search that can be performed more efficiently using historical data of where an object is frequently placed. Our model can accurately identify the complete 3D location of a hidden object on the first try significantly faster than a naive random search. These extensions to semantic abstraction hope to provide household robots with the skills necessary to save time and effort when looking for lost objects.