From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle

TL;DR

This paper presents Hive, a multi-modal, knowledge-aware planning system that uses LLMs and PDDL to generate explainable, complex action sequences for real-world queries, outperforming existing methods.

Contribution

Introduces Hive, a novel multi-modal agent system leveraging LLMs and PDDL for explainable, constraint-aware planning of atomic actions across diverse models.

Findings

Hive outperforms existing systems in task selection accuracy.

The MuSE benchmark effectively evaluates multi-modal agent capabilities.

Hive guarantees explainability and user constraint adherence in complex tasks.

Abstract

In response to the call for agent-based solutions that leverage the ever-increasing capabilities of the deep models' ecosystem, we introduce Hive -- a comprehensive solution for knowledge-aware planning of a set of atomic actions to address input queries and subsequently selecting appropriate models accordingly. Hive operates over sets of models and, upon receiving natural language instructions (i.e. user queries), schedules and executes explainable plans of atomic actions. These actions can involve one or more of the available models to achieve the overall task, while respecting end-users specific constraints. Notably, Hive handles tasks that involve multi-modal inputs and outputs, enabling it to handle complex, real-world queries. Our system is capable of planning complex chains of actions while guaranteeing explainability, using an LLM-based formal logic backbone empowered by PDDL operations. We introduce the MuSE benchmark in order to offer a comprehensive evaluation of the multi-modal capabilities of agent systems. Our findings show that our framework redefines the state-of-the-art for task selection, outperforming other competing systems that plan operations across multiple models while offering transparency guarantees while fully adhering to user constraints.