ALRM: Agentic LLM for Robotic Manipulation

TL;DR

This paper introduces ALRM, a modular agentic framework that enhances robotic manipulation by integrating large language models with reasoning loops, supporting diverse instructions and systematic evaluation through a new benchmark.

Contribution

ALRM is the first framework combining agentic reasoning with robotic control, enabling multistep planning, reflection, and revision using LLMs in a modular, interpretable manner.

Findings

Claude-4.1-Opus performs best among closed-source models.

Falcon-H1-7B excels among open-source models.

ALRM achieves reliable robotic execution across diverse tasks.

Abstract

Large Language Models (LLMs) have recently empowered agentic frameworks to exhibit advanced reasoning and planning capabilities. However, their integration in robotic control pipelines remains limited in two aspects: (1) prior \ac{llm}-based approaches often lack modular, agentic execution mechanisms, limiting their ability to plan, reflect on outcomes, and revise actions in a closed-loop manner; and (2) existing benchmarks for manipulation tasks focus on low-level control and do not systematically evaluate multistep reasoning and linguistic variation. In this paper, we propose Agentic LLM for Robot Manipulation (ALRM), an LLM-driven agentic framework for robotic manipulation. ALRM integrates policy generation with agentic execution through a ReAct-style reasoning loop, supporting two complementary modes: Code-asPolicy (CaP) for direct executable control code generation, and Tool-as-Policy (TaP) for iterative planning and tool-based action execution. To enable systematic evaluation, we also introduce a novel simulation benchmark comprising 56 tasks across multiple environments, capturing linguistically diverse instructions. Experiments with ten LLMs demonstrate that ALRM provides a scalable, interpretable, and modular approach for bridging natural language reasoning with reliable robotic execution. Results reveal Claude-4.1-Opus as the top closed-source model and Falcon-H1-7B as the top open-source model under CaP.