Enhancing Robustness in Language-Driven Robotics: A Modular Approach to Failure Reduction

TL;DR

This paper introduces a modular architecture for LLM-driven robotics that improves task robustness and success rates by addressing grounding issues and enabling real-time error recovery, especially with smaller, efficient language models.

Contribution

The paper presents a novel modular system with an expected outcomes module and feedback mechanism to enhance robustness in LLM-based robotic planning and execution.

Findings

Significant improvement in task success rates in simulation and real robots.

Effective real-time error recovery mechanism demonstrated.

Smaller LLMs can be used efficiently for robust robotic tasks.

Abstract

Recent advances in large language models (LLMs) have led to significant progress in robotics, enabling embodied agents to better understand and execute open-ended tasks. However, existing approaches using LLMs face limitations in grounding their outputs within the physical environment and aligning with the capabilities of the robot. This challenge becomes even more pronounced with smaller language models, which are more computationally efficient but less robust in task planning and execution. In this paper, we present a novel modular architecture designed to enhance the robustness of LLM-driven robotics by addressing these grounding and alignment issues. We formalize the task planning problem within a goal-conditioned POMDP framework, identify key failure modes in LLM-driven planning, and propose targeted design principles to mitigate these issues. Our architecture introduces an ``expected outcomes'' module to prevent mischaracterization of subgoals and a feedback mechanism to enable real-time error recovery. Experimental results, both in simulation and on physical robots, demonstrate that our approach significantly improves task success rates for pick-and-place and manipulation tasks compared to both larger LLMs and standard baselines. Through hardware experiments, we also demonstrate how our architecture can be run efficiently and locally. This work highlights the potential of smaller, locally-executable LLMs in robotics and provides a scalable, efficient solution for robust task execution.