HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models

TL;DR

This study systematically investigates hallucinations in large language model-driven embodied agents, revealing their causes, triggers, and limitations in handling scene-task inconsistencies during complex navigation tasks.

Contribution

It introduces a hallucination probing set for LLM-based embodied agents and provides a comprehensive evaluation of 12 models across simulation environments.

Findings

Models exhibit reasoning but fail to resolve scene-task inconsistencies

Hallucination rates can be increased up to 40x with the probing set

Fundamental limitations in handling infeasible tasks are identified

Abstract

Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene-task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40x higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies-highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies.