MultiTalk: Introspective and Extrospective Dialogue for Human-Environment-LLM Alignment

TL;DR

MultiTalk introduces a dialogue-based framework for improving LLM-driven task planning in robots by enabling introspective and extrospective interactions, which ground plans in environmental and capability contexts and resolve ambiguities.

Contribution

The paper presents MultiTalk, a novel LLM-based planning method using dialogue loops to enhance plan accuracy and robustness in embodied agent tasks.

Findings

Outperforms baseline methods in robotic manipulation tasks.

Improves plan reliability through dialogue-based grounding.

Demonstrates robustness via ablation studies.

Abstract

LLMs have shown promising results in task planning due to their strong natural language understanding and reasoning capabilities. However, issues such as hallucinations, ambiguities in human instructions, environmental constraints, and limitations in the executing agent's capabilities often lead to flawed or incomplete plans. This paper proposes MultiTalk, an LLM-based task planning methodology that addresses these issues through a framework of introspective and extrospective dialogue loops. This approach helps ground generated plans in the context of the environment and the agent's capabilities, while also resolving uncertainties and ambiguities in the given task. These loops are enabled by specialized systems designed to extract and predict task-specific states, and flag mismatches or misalignments among the human user, the LLM agent, and the environment. Effective feedback pathways between these systems and the LLM planner foster meaningful dialogue. The efficacy of this methodology is demonstrated through its application to robotic manipulation tasks. Experiments and ablations highlight the robustness and reliability of our method, and comparisons with baselines further illustrate the superiority of MultiTalk in task planning for embodied agents.