Architecting Large Action Models for Human-in-the-Loop Intelligent Robots

TL;DR

This paper demonstrates that large action models for robots can be built by combining existing foundation models with symbolic verification, enabling reliable, interpretable, and human-in-the-loop robotic systems without extensive training.

Contribution

It introduces a neuro-symbolic approach to large action models, integrating off-the-shelf models with symbolic wrappers and verification for improved control and safety.

Findings

Effective integration of perception models with logic-driven core.

Planning with PDDL enables human-in-the-loop verification.

Reliable, verifiable robotic action generation achieved.

Abstract

The realization of intelligent robots, operating autonomously and interacting with other intelligent agents, human or artificial, requires the integration of environment perception, reasoning, and action. Classic Artificial Intelligence techniques for this purpose, focusing on symbolic approaches, have long-ago hit the scalability wall on compute and memory costs. Advances in Large Language Models in the past decade (neural approaches) have resulted in unprecedented displays of capability, at the cost of control, explainability, and interpretability. Large Action Models aim at extending Large Language Models to encompass the full perception, reasoning, and action cycle; however, they typically require substantially more comprehensive training and suffer from the same deficiencies in reliability. Here, we show it is possible to build competent Large Action Models by composing off-the-shelf foundation models, and that their control, interpretability, and explainability can be effected by incorporating symbolic wrappers and associated verification on their outputs, achieving verifiable neuro-symbolic solutions for intelligent robots. Our experiments on a multi-modal robot demonstrate that Large Action Model intelligence does not require massive end-to-end training, but can be achieved by integrating efficient perception models with a logic-driven core. We find that driving action execution through the generation of Planning Domain Definition Language (PDDL) code enables a human-in-the-loop verification stage that effectively mitigates action hallucinations. These results can support practitioners in the design and development of robotic Large Action Models across novel industries, and shed light on the ongoing challenges that must be addressed to ensure safety in the field.