CommCP: Efficient Multi-Agent Coordination via LLM-Based Communication with Conformal Prediction

TL;DR

CommCP is a novel multi-agent communication framework using LLMs and conformal prediction to improve coordination and efficiency in complex embodied question answering tasks in household environments.

Contribution

We introduce CommCP, a decentralized LLM-based communication method with conformal prediction for multi-agent embodied question answering, addressing coordination and information gathering challenges.

Findings

Significantly improves task success rate in MM-EQA tasks.

Enhances exploration efficiency in complex household scenarios.

Outperforms baseline communication methods.

Abstract

To complete assignments provided by humans in natural language, robots must interpret commands, generate and answer relevant questions for scene understanding, and manipulate target objects. Real-world deployments often require multiple heterogeneous robots with different manipulation capabilities to handle different assignments cooperatively. Beyond the need for specialized manipulation skills, effective information gathering is important in completing these assignments. To address this component of the problem, we formalize the information-gathering process in a fully cooperative setting as an underexplored multi-agent multi-task Embodied Question Answering (MM-EQA) problem, which is a novel extension of canonical Embodied Question Answering (EQA), where effective communication is crucial for coordinating efforts without redundancy. To address this problem, we propose CommCP, a novel LLM-based decentralized communication framework designed for MM-EQA. Our framework employs conformal prediction to calibrate the generated messages, thereby minimizing receiver distractions and enhancing communication reliability. To evaluate our framework, we introduce an MM-EQA benchmark featuring diverse, photo-realistic household scenarios with embodied questions. Experimental results demonstrate that CommCP significantly enhances the task success rate and exploration efficiency over baselines. The experiment videos, code, and dataset are available on our project website: https://comm-cp.github.io.