$\mathcal{P}^3$: Toward Versatile Embodied Agents

TL;DR

The paper introduces $ ext{P}^3$, a versatile embodied agent framework that actively perceives environments, flexibly uses tools without feedback, and dynamically plans multi-task execution, enhancing real-world adaptability.

Contribution

The paper presents $ ext{P}^3$, a novel unified framework integrating real-time perception and dynamic scheduling for embodied agents, addressing key challenges in versatility and multi-task management.

Findings

Achieves high transferability in real-world environments.

Enables tool use without feedback, increasing flexibility.

Improves multi-task planning with dynamic prioritization.

Abstract

Embodied agents have shown promising generalization capabilities across diverse physical environments, making them essential for a wide range of real-world applications. However, building versatile embodied agents poses critical challenges due to three key issues: dynamic environment perception, open-ended tool usage, and complex multi-task planning. Most previous works rely solely on feedback from tool agents to perceive environmental changes and task status, which limits adaptability to real-time dynamics, causes error accumulation, and restricts tool flexibility. Furthermore, multi-task scheduling has received limited attention, primarily due to the inherent complexity of managing task dependencies and balancing competing priorities in dynamic and complex environments. To overcome these challenges, we introduce $\mathcal{P}^3$, a unified framework that integrates real-time perception and dynamic scheduling. Specifically, $\mathcal{P}^3$ enables 1) \textbf Perceive relevant task information actively from the environment, 2) \textbf Plug and utilize any tool without feedback requirement, and 3) \textbf Plan multi-task execution based on prioritizing urgent tasks and dynamically adjusting task order based on dependencies. Extensive real-world experiments show that our approach bridges the gap between benchmarks and practical deployment, delivering highly transferable, general-purpose embodied agents. Code and data will be released soon.