Language-Enhanced Mobile Manipulation for Efficient Object Search in Indoor Environments

TL;DR

This paper presents GODHS, a language-enhanced hierarchical navigation framework that integrates semantic perception and spatial reasoning, enabling robots to search for objects more efficiently in indoor environments.

Contribution

It introduces a novel framework combining large language models with hierarchical decision-making and a heuristic motion planner for improved object search.

Findings

GODHS outperforms traditional search strategies in efficiency.

Semantic reasoning improves search accuracy.

Framework validated in Isaac Sim environment.

Abstract

Enabling robots to efficiently search for and identify objects in complex, unstructured environments is critical for diverse applications ranging from household assistance to industrial automation. However, traditional scene representations typically capture only static semantics and lack interpretable contextual reasoning, limiting their ability to guide object search in completely unfamiliar settings. To address this challenge, we propose a language-enhanced hierarchical navigation framework that tightly integrates semantic perception and spatial reasoning. Our method, Goal-Oriented Dynamically Heuristic-Guided Hierarchical Search (GODHS), leverages large language models (LLMs) to infer scene semantics and guide the search process through a multi-level decision hierarchy. Reliability in reasoning is achieved through the use of structured prompts and logical constraints applied at each stage of the hierarchy. For the specific challenges of mobile manipulation, we introduce a heuristic-based motion planner that combines polar angle sorting with distance prioritization to efficiently generate exploration paths. Comprehensive evaluations in Isaac Sim demonstrate the feasibility of our framework, showing that GODHS can locate target objects with higher search efficiency compared to conventional, non-semantic search strategies. Website and Video are available at: https://drapandiger.github.io/GODHS

Tables

Table 1. Table II : Comparison of Sorting Methods for EE and CH Poses.

Sorting Method	EE Ratio	CH Ratio	Time Ratio
Unoptimized	2.81	2.37	1.00
EE Sorting	1.75	2.41	0.87
CH Sorting	2.79	1.60	0.83
Both Optimized	1.77	1.59	0.66

Equations

$${\mathbf{r}}^{\ast }=\underset{\mathsf{r}\in \mathcal{K}\mathcal{B}}{\mathrm{argmax}}P\left(\mathsf{r}\mid \mathcal{O}\right),$$

$$\mathbf{R}=\underset{\mathsf{r}\in \mathcal{R}}{\mathrm{argsort}}P(\mathsf{r}\mid \mathcal{R},\mathsf{t}),$$

$$\mathbf{C}=\underset{\mathsf{c}\in \mathcal{C}}{\mathrm{argsort}}P(\mathsf{c}\mid \mathcal{C},\mathsf{t}).$$

$$\hskip-1.99997pt\mathcal{M}_{F}^{\text{top}}=\left\{(x_{F},y_{F},z_{F})\,\Big{|}\,\begin{array}[]{l}(x_{F},y_{F})\in\mathcal{M}_{C}^{*},\\ (x_{F},y_{F},z_{F}^{\prime})\in\mathcal{M}_{C},\\ z_{F}=\\ \max\{z_{F}^{\prime}\mid(x_{F},y_{F},z_{F}^{\prime})\}\end{array}\right\}.$$

$$\mathcal{M}_{F}^{\text{sides}}=\left\{(x_{F},y_{F},z_{F})\,\Big{|}\,\begin{array}[]{l}(x_{F},y_{F})\in\partial\mathcal{M}_{C}^{*},\\ (x_{F},y_{F},z_{F}^{\prime})\in\mathcal{M}_{C},\\ z_{F}\in\\ \hskip-8.00003pt\left[0,\max\{z_{F}^{\prime}\mid(x_{F},y_{F},z_{F}^{\prime})\}\right]\end{array}\right\}.$$

$$\mathcal{M}_{F}^{\text{bottom}}=\left\{(x_{F},y_{F},z_{F0})\,\Big{|}\,(x_{F},y_{F})\in\mathcal{M}_{C}^{*}\right\}.$$

$$\rho ={\arctan }_2\left(y_{\text{CH}}-\bar{y},x_{\text{CH}}-\bar{x}\right),$$

$$OSR=w_1\cdot R_r+w_2\cdot R_c+w_3\cdot R_i,$$

Full text

I Introduction

When humans search for objects in unfamiliar environments, they rely on a hierarchical understanding of semantic information to rapidly localize potential object placements [1]. Inspired by this ability, our approach emulates the human strategy of leveraging semantic cues—e.g., “pillows often lie on beds”—to guide the search process. Rather than exhaustively scanning an entire space, humans focus on “carriers” (like beds or tables) that are most likely to hold a target object, significantly reducing the search effort. Mobile manipulation has made great strides in recent years [2]. However, most robots still rely on purely spatial exploration strategies and ignore semantic cues, which makes them inefficient in unfamiliar environments.

In addition, current navigation systems exhibit two major shortcomings. First, although they may incorporate basic semantic labels, they often fail to reason about the relationships between known objects and unknown targets. Second, while vision-language models (VLMs) can align language with image features, they often lack the broad commonsense knowledge that large language models (LLMs) possess. In principle, LLMs combined with robust semantic segmentation could provide deeper real-world contextual reasoning.

To address these gaps, this paper introduces the Goal-Oriented Dynamically Heuristic-Guided Hierarchical Search (GODHS) framework, an exploratory study inspired by human search behavior. Building on the notion that people mentally decompose a search task—from room, to carrier, to specific features—GODHS employs LLMs to orchestrate a five-level search hierarchy: scene $\rightarrow$ room $\rightarrow$ carrier $\rightarrow$ feature $\rightarrow$ item. As illustrated in Fig. 1, each level narrows the search scope through an LLM-driven reasoning process.

Implementing this hierarchical pipeline presents two primary challenges that our work addresses:

• •

Reliable Hierarchical Reasoning: The system must translate a high-level search goal into a multi-step, semantically-grounded plan. GODHS achieves this by leveraging LLMs to hierarchically decompose the problem. To ensure the reliability of the LLM’s guidance, we employ structured prompts and logical constraints at each level of the hierarchy.

• •

Efficient Physical Exploration: A mobile manipulator exploring a carrier faces a vast action space for positioning its base and end-effector. We propose a heuristic-based motion planner that combines polar and lexicographical sorting to efficiently generate and prioritize exploration poses, significantly reducing redundant travel and execution time.

II Related Work

Robotic object search has progressed through three main paradigms. Early geometric approaches established principles but faced critical limitations. These methods constrained search regions using spatial affordances [3] or modeled uncertainty with probabilistic frameworks [4, 5, 6]. However, they were often effective only in structured environments, as they required exhaustive object relationship priors and could not infer latent functional associations (e.g., that medicine belongs in a cabinet). Underpinning these systems are foundational motion planning algorithms [7], which have evolved from sampling-based approaches to adapted high-DoF robotic systems for real-time planning [8, 9]. Furthermore, traditional decision-making frameworks like POMDPs [10] remained confined to predefined taxonomies, limiting their adaptability to novel objects.

The advent of VLMs enabled open-vocabulary object recognition but introduced new constraints. While semantic mapping systems [11] enriched spatial maps with object labels, they often treated semantics as static attributes, overlooking dynamic relational reasoning. Other works used hierarchical graphs to model object interactions [12], but these rigid structures typically required complete environmental pre-knowledge. Modern VLM-integrated navigation methods [13, 14] can locate open-vocabulary targets but may overfit to superficial visual-text correlations [15], struggling to chain contextual relationships for multi-step reasoning.

Recent integrations of LLMs show strong potential but still face challenges. LLM-augmented planners generate plausible search hypotheses [16], yet many rely on flat decision structures that inefficiently evaluate object relationships. Scene-graph methods capture relationships more dynamically [17, 18] but often lack incremental refinement and assume high observability [19]. While progress has been made in language grounding, few methods emulate human-like hierarchical task decomposition. Our work bridges this divide by synergizing LLM-driven commonsense reasoning with principled hierarchical action planning, avoiding both the inflexibility of purely geometric heuristics and the inefficiency of flat neural architectures.

III Methodology

Our search approach follows a logical hierarchical progression to locate the ultimate target. We construct a hierarchically expandable algorithm, GODHS, that integrates environmental perception with commonsense reasoning from a large language model (LLM) to derive semantic action sequences(Sec. III-A). To ensure the reliability of the LLM-driven reasoning, we employ structured prompts and constraints at each stage of the hierarchy. To ensure the mobile robot advances sequentially around the carrier, we employ dictionary mapping and polar angle sorting (Sec. III-B).

III-A GODHS Framework

Goal-Oriented Dynamically Heuristic-Guided Hierarchical Search is an efficient search approach that combines cognitive reasoning, sensory data processing, and decision-making strategies. The detailed process is shown in Figure 2. The GODHS algorithm is designed primarily to locate target objects in complex environments. It has four key features:

• •

Goal-Oriented: The search is consistently focused on the final target, allowing the system to effectively prune the search space at each level of the hierarchy.

• •

Dynamically Updated: The system dynamically reassesses and reorders search priorities based on new information gathered during execution, inspired by dynamic tree search structures [20].

• •

Heuristic-Guided: The process is guided by contextual probabilities from the LLM, which infers likely search locations based on commonsense knowledge rather than strict mathematical optimization.

• •

Bounded Hierarchical: The search space is organized into a five-level hierarchy (Scene $\rightarrow$ Room $\rightarrow$ Carrier $\rightarrow$ Feature $\rightarrow$ Item), progressively narrowing the scope to avoid an exhaustive global search [21].

The full process is detailed in Algorithm 1, which takes a scene name $\mathsf{s}$ and a target $\mathsf{t}$ as input. The algorithm initializes the required maps ($\mathcal{M_{S}},\mathcal{M_{R}},\mathcal{M_{C}}$) and lists ($\mathbf{R},\mathbf{C},\mathbf{F}$). The process begins with an autonomous exploration phase (Lines 2-8), where the robot constructs a global scene map $\mathcal{M_{S}}$ by sequentially visiting and mapping all accessible rooms. For each room, a local map $\mathcal{M_{R}}$ is built using LiDAR data and integrated into the global map: $\mathcal{M}_{S}\leftarrow\mathcal{M}_{S}\cup\left(\mathcal{M}_{R}\setminus\mathcal{M}_{S}\right)$.

The function $\text{InferRoom}(\cdot)$ in Line 7 employs an LLM to predict the room category $\mathbf{r}^{*}$ based on the set of observed objects $\mathcal{O}$:

[TABLE]

where $\mathcal{KB}$ represents the commonsense knowledge of the pre-trained LLM. This generates a room-to-map correspondence $\mathcal{I_{R}}:\mathcal{R}\to\mathcal{M_{R}}$. Finally, the language model ranks elements of $\mathcal{R}$ by likelihood of target $\mathsf{t}$ in $\mathcal{R}$:

[TABLE]

The agent navigates to rooms according to the order in $\mathbf{R}$. Within each room, the $\text{ClassifyCarrier}(\cdot)$ function in Line 11 prompts the LLM to identify which of the observed objects are plausible ‘carriers’ for the target $\mathsf{t}$. This step directly yields a filtered list of carrier objects $\mathcal{C}\subseteq\mathcal{O}$ without relying on an explicit numerical threshold. Focused scanning to capture carrier point clouds $\mathcal{M}_{C}$ is performed and carrier-to-map correspondence $\mathcal{I}_{C}:\mathcal{C}\rightarrow\mathcal{M}_{C}$ will be established. These identified carriers are then ranked by the LLM based on their relevance to the target $\mathsf{t}$:

[TABLE]

Guided by the prioritized carrier list $\mathbf{C}$, the agent sequentially inspects each carrier $\mathsf{c}\in\mathbf{C}$. At the Feature Level, the agent must determine which specific parts of the carrier are most relevant for finding the target $\mathsf{t}$. We prompt the LLM to perform a direct selection and ranking task. Given a set of spatial regions $\mathcal{F}=\{\text{`top'},\text{`bottom'},\text{`sides'},\text{`inside'}\}$, the LLM is tasked to return an ordered list of the most plausible features to inspect for the given carrier and target. This process yields a final, prioritized sequence of features $\mathbf{F}=\underset{\mathsf{f}\in\mathcal{F}}{\text{argsort}}\,P(\mathsf{f}\mid\mathcal{F},\mathsf{t})$. The final stage of the algorithm 1 in Lines 17-28 then iterates through this robustly generated list $\mathbf{F}$, creating and executing motion plans to visually inspect each feature in order, until the target is found.

To ensure the reliability of the LLM’s guidance within the GODHS framework, we must address the model’s inherent tendency for statistical hallucination [22]. Inspired by recent structured reasoning techniques like Chain-of-Thought [23] and self-refinement [24], we implement a multi-stage verification process for LLM queries. This process, illustrated in Fig. 3 (Left), involves cleaning and structuring the input, executing the core reasoning task, and correcting the output to ensure it is semantically consistent and syntactically valid for the robot.

The cornerstone of this approach is a carefully structured prompt design, as shown in Fig. 3 (Right). For example, to determine the searchable features for a ‘fridge’ potentially containing an ‘orange’, the prompt explicitly defines the task, constrains the possible outputs to a predefined set (‘top’, ‘bottom’, ‘sides’, ‘inside’), provides clarifying examples (e.g., a ‘bathtub’ should return ‘top’), and enforces a strict, machine-readable output format. This structured prompting is crucial for grounding the LLM’s abstract knowledge to the specific, operational needs of the robot at each level of the hierarchy.

III-B Heuristic-Based Pose Generation and Sorting

This subsection details our heuristic-based methodology for computing chassis (CH) and end-effector (EE) poses. The goal is to efficiently generate structured search trajectories that enable the robot to systematically inspect a carrier’s features while reducing redundant motion. The process involves five sequential stages: (1) extraction of geometric features $\mathcal{M}_{F}$ from point cloud data, (2) generation and selection of EE poses $\mathcal{P_{EE}}$, (3) fast evaluation and selection of chassis poses $\mathcal{P_{CH}}$ via geometric solution, (4) validation of the relation between CH and EE $\mathcal{P_{CH}^{EE}}$ with subsequent inverse kinematics (IK) solving, and (5) sorting of the resulting pose mappings $\mathcal{P_{\text{CH}}^{\text{EE}}}$ for sequential execution.

The process begins with the acquisition of the point cloud of single carrier $\mathcal{M}_{C}\subset\mathbb{R}^{3}$ and the occupied grid of the carrier $\mathcal{M}_{C}^{*}\subset\mathbb{R}^{2}$. These data sources are used to extract the carrier’s relevant features, collectively denoted as $\mathcal{M}_{F}$. The features are subdivided into four components: Top Surface, Side Surface, Bottom Area and Inside Area.

The top surface feature $\mathcal{M}_{F}^{\text{top}}$ is extracted directly from point cloud $\mathcal{M}_{C}$. For each $(x_{F},y_{F})$ coordinate in the grid, the top surface point is identified as the point with the maximum $z_{F}$ value. Formally, if we denote by $\mathcal{M}_{C}^{*}$ the grid cell corresponding to a particular $(x_{F},y_{F})$ location, then

[TABLE]

The side surface feature $\mathcal{M}_{F}^{\text{sides}}$ is derived from the boundary of the volumetric occupied grid $\mathcal{M}_{C}^{*}$. For each $(x_{F},y_{F})$ coordinate lying on the topologically defined edge of $\mathcal{M}_{C}^{*}$ (denoted by $\partial\mathcal{M}_{C}^{*}$), the corresponding side surface value is determined by extracting the maximum $z_{F}$ value in $\mathcal{M}_{C}$ over the range starting from $z_{F}=0$. This could be expressed as:

[TABLE]

This representation captures the vertical boundaries of the carrier. The bottom area feature $\mathcal{M}_{F}^{\text{bottom}}$ is defined by the intersection of the occupied grid $\mathcal{M}_{C}^{*}$ with a spatially constrained horizontal plane at a fixed height $z_{F}=z_{F0}$:

[TABLE]

The inside feature is identified using a dedicated geometric analysis procedure designed to detect potential openings or enclosed spaces on the carrier.

For the mobile manipulator, two types of poses are essential: the chassis pose and the EE pose. The candidate chassis pose is defined as $\mathbf{p}_{CH}^{Can}=(x_{CH},y_{CH},\theta_{CH})$, where $(x_{CH},y_{CH})$ represents the base position and $\theta_{CH}$ the yaw angle. The candidate EE pose is defined as $\mathbf{p}_{EE}^{Can}=(x_{EE},y_{EE},z_{EE},\phi_{EE},\theta_{EE},\psi_{EE})$. The direction vector of the camera’s view can be expressed as $\mathbf{v}_{\mathrm{dir}}=(\cos(\theta_{EE})\cdot\cos(\psi_{EE}),\cos(\theta_{EE})\cdot\sin(\psi_{EE}),\sin(\theta_{EE}))$. The vector pointing from the camera to the surface point is $\mathbf{v}_{\mathrm{point}}=\begin{pmatrix}x_{F}-x_{EE},y_{F}-y_{EE},z_{F}-z_{EE}\end{pmatrix}$. The surface point is considered covered by the vision cone of the camera, if the conditions of horizontal and vertical angle $\theta_{\text{horizontal}}=\arccos\left(\frac{v_{\text{dir},x}\cdot v_{\text{point},x}+v_{\text{dir},z}\cdot v_{\text{point},z}}{\|v_{\text{dir}}\|\|v_{\text{point}}\|}\right)<\text{FoV}_{\text{horizontal}}/2$ and $\theta_{\text{vertical}}=\arccos\left(\frac{v_{\text{dir},y}\cdot v_{\text{point},y}+v_{\text{dir},z}\cdot v_{\text{point},z}}{\|v_{\text{dir}}\|\|v_{\text{point}}\|}\right)<\text{FoV}_{\text{vertical}}/2$ are satisfied, and the candidate pose earns one point. We employ a greedy algorithm [25] as a heuristic to select a set of EE poses $\mathcal{P_{EE}}$ that provide sufficient visual coverage.

The set of chassis poses $\mathcal{P_{CH}}$ is selected from $\mathbf{p}_{CH}^{Can}$ via deterministic greedy algorithm based on $\mathcal{P_{EE}}$. If a geometric solution can establish a collision-free connection, one point is awarded. The selection continues until $\mathcal{P_{CH}}$ is found that spatially covers all elements in $\mathcal{P_{EE}}$.

Since fast-solving cannot guarantee a feasible solution for the actual robot operation, and directly solving the IK for all poses would lead to excessive computational overhead, we perform IK solving with Levenberg–Marquardt algorithm [26] for each chassis pose $\mathbf{P_{\text{CH}}}$ in $\mathcal{P_{CH}}$ and each end-effector pose $\mathbf{P_{\text{EE}}}$ in $\mathcal{P_{EE}}$. If numerically stable solutions exist, the mapping is added to the dictionary $\mathcal{P_{CH}^{EE}}$.

The dictionary $\mathcal{P_{CH}^{EE}}$ is non-sequential, so a sorting method is required to obtain an ordered version $\mathcal{P}_{\text{CH}}^{\text{EE}}$. For each end-effector pose corresponding to a chassis pose, we apply Lexicographical Sorting based on the sequence $z_{EE},y_{EE},x_{EE},\psi_{EE},\theta_{EE},\phi_{EE}$.

For all chassis poses, we do not simply sort based on spatial distance but instead ensure movement proceeds clockwise around the carrier. To achieve this, we use centroid-aligned Polar Angle Sorting [27], and the geometric centroid $(\bar{x},\bar{y})=\left(\frac{1}{n}\sum_{i=1}^{n}x_{i},\frac{1}{n}\sum_{i=1}^{n}y_{i}\right)$ is computed where $(x_{i},y_{i})\in\mathcal{M}_{C}$, with $n$ being the size of the point set. Then we can calculate its angle relative to the average point by:

[TABLE]

where $(x_{\text{CH}},y_{\text{CH}})$ is the position of each chassis. Following the angle $\rho$, the chassis poses will be sorted. Using this method, the movement platform can advance around the carrier while steadily maintaining its direction of movement, avoiding the situation where sorting based on the nearest spatial distance might result in the actual path distance being significantly greater than the spatial distance.

IV Experimental Setup and Results

IV-A Evaluation system setup

The simulation framework is developed using NVIDIA Isaac Sim, offering photorealistic sensor data and accurate physics simulation. The robotic platform is DARKO, featuring an omnidirectional RB-KAIROS base and a 7-DOF Franka Emika Panda arm, integrated with an Ouster OS1 LiDAR and an Intel RealSense D435 camera. The system runs on ROS Noetic, employing the standard Navigation Stack for base control and MoveIt for manipulator planning. A local Ollama server powers the cognitive layer for LLM-based interaction.

IV-B Feasibility Testing

The feasibility test evaluates the system’s target search and semantic planning capabilities in a constrained indoor environment. Experiments were conducted in a simulated “flat” scene with seven functional zones (e.g., kitchen, living room, bedroom), as shown in Fig. 5. A representative task involves locating an “orange” placed inside a fridge, requiring the robot to navigate across rooms and identify relevant carriers and features. The process begins with environment exploration to build a complete occupancy map. Semantic segmentation detects objects, which are then processed by the LLM to infer room categories (e.g., recognizing a bed and dressing table suggests a bedroom). Based on the query “orange,” the LLM prioritizes searching the living room followed by the kitchen. In the living room, the robot inspects the coffee table’s top and bottom but finds nothing. It then moves to the kitchen, identifies the fridge as the most likely carrier, and successfully locates the orange inside.

To evaluate the search efficiency of our approach, we conducted 81 experiments using Qwen2.5-7B and GPT-4o. We define search efficiency metrics: the Room Search Rate ($R_{r}$), Carrier Search Rate ($R_{c}$), and Item Search Rate ($R_{i}$), which represent the percentage of rooms, carriers, or items, respectively, that were searched before the target was found. A lower value for these metrics indicates a more efficient search. The Overall Search Rate (OSR) is a weighted average of these values, representing the overall search cost:

[TABLE]

where $w_{1}$, $w_{2}$, and $w_{3}$ are weights that reflect the relative importance of each search level. A reasonable choice for the weights could be assigned as $w_{1}=0.2$, $w_{2}=0.3$, $w_{3}=0.5$.

We compare our method against two traditional non-semantic search methods: a full Coverage Search and a Random Walk search. The results are shown in Table I.

The results in Table I validate the efficiency of the GODHS framework. Guided by both LLMs, our system demonstrates significantly lower search rates across all categories compared to the non-semantic baselines. This indicates that by leveraging hierarchical, semantic guidance, the robot needs to explore a much smaller fraction of the environment to locate the target, confirming that the approach significantly reduces search cost and improves efficiency.

We identified three categories of failure modes: (i) hardware limitations—for example, the manipulator cannot access areas close to the floor; (ii) insufficient common-sense in the LLM—for instance, it may instruct the robot to inspect a non-openable exterior panel of a fridge; and (iii) semantic ambiguity—such as misclassifying a billiard table.

IV-C Performance Evaluation

In this section, we evaluate the performance of our heuristic-based motion planner, specifically the pose sorting strategies detailed in Sec. III-B. The experiment is designed to assess the effectiveness of our approach in optimizing the robot’s exploration path and reducing execution time in the simulated environment.

To systematically assess the effectiveness of our proposed sorting strategies, we compare four configurations: an unoptimized baseline with no sorting applied to either EE or CH poses; a configuration where only EE poses are optimized via lexicographical sorting; another where only CH poses are optimized using polar angle sorting; and a final setup where both optimizations are applied together.

We evaluate three key performance metrics: the Normalized EE Path Length, which represents the ratio of the total EE travel distance to the theoretical shortest path; the Normalized CH Path Length, defined similarly for the CH poses; and the Execution Time Ratio, which is the ratio of execution time after optimization to that of the unoptimized case, with a lower value indicating better efficiency.

Table II presents the comparative results. The data demonstrates that applying lexicographical sorting to EE poses and polar angle sorting to CH poses independently yield significant improvements in their respective path lengths. When both strategies are combined, the system achieves the highest optimization in both path efficiency and overall execution time, confirming the effectiveness of our heuristic motion planning strategy.

V Conclusion

In this work, we presented the GODHS Framework, which integrates an LLM’s commonsense reasoning with a multi-level decision process to improve search efficiency. This is achieved by using structured prompts to ensure reliable reasoning and a heuristic-based motion planner with pose sorting to generate efficient exploration trajectories. Experiments conducted in simulation demonstrated the feasibility of our approach and more efficient search performance compared to non-semantic strategies. Future work will focus on deploying the framework on a physical robot and exploring the integration of multi-modal foundation models [28] and social navigation models [29] to enhance its capabilities.