LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments

TL;DR

LEGS-POMDP is a modular system that combines language, gesture, and visual data within a POMDP framework to improve open-world object search under uncertainty, demonstrated through simulation and real-world experiments.

Contribution

It introduces a novel POMDP-based approach that explicitly models uncertainty over object identity and location, integrating multimodal inputs for robust open-world object search.

Findings

Achieved 89% success rate in simulation environments.

Multimodal fusion outperforms unimodal baselines.

Validated system on a quadruped robot in real-world scenarios.

Abstract

To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89\% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.