Multi-LLM QA with Embodied Exploration

TL;DR

This paper investigates how multi-agent LLM systems can perform question-answering in unknown environments through embodied exploration, introducing a new aggregation method that significantly improves accuracy.

Contribution

It introduces Multi-Embodied LLM Explorers (MELE) for QA in unknown environments and proposes a learned aggregation method that outperforms traditional approaches.

Findings

CAM achieves 46% higher accuracy than other methods.

Multiple LLM agents can effectively explore and answer questions about household environments.

The study provides code and datasets for further research.

Abstract

Large language models (LLMs) have grown in popularity due to their natural language interface and pre trained knowledge, leading to rapidly increasing success in question-answering (QA) tasks. More recently, multi-agent systems with LLM-based agents (Multi-LLM) have been utilized increasingly more for QA. In these scenarios, the models may each answer the question and reach a consensus or each model is specialized to answer different domain questions. However, most prior work dealing with Multi-LLM QA has focused on scenarios where the models are asked in a zero-shot manner or are given information sources to extract the answer. For question answering of an unknown environment, embodied exploration of the environment is first needed to answer the question. This skill is necessary for personalizing embodied AI to environments such as households. There is a lack of insight into whether a Multi-LLM system can handle question-answering based on observations from embodied exploration. In this work, we address this gap by investigating the use of Multi-Embodied LLM Explorers (MELE) for QA in an unknown environment. Multiple LLM-based agents independently explore and then answer queries about a household environment. We analyze different aggregation methods to generate a single, final answer for each query: debating, majority voting, and training a central answer module (CAM). Using CAM, we observe a $46\%$ higher accuracy compared against the other non-learning-based aggregation methods. We provide code and the query dataset for further research.