Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation

TL;DR

This paper introduces Act-Observe-Rewrite (AOR), a framework where multimodal language models improve robot manipulation by generating and rewriting executable code based on visual observations and outcomes, enabling failure diagnosis and policy refinement.

Contribution

AOR uniquely leverages interpretable code as the policy, allowing in-context learning and failure diagnosis without demonstrations or gradient updates.

Findings

High success rates across three manipulation tasks

Effective failure diagnosis and policy rewriting

No need for demonstrations or reward engineering

Abstract

Can a multimodal language model learn to manipulate physical objects by reasoning about its own failures-without gradient updates, demonstrations, or reward engineering? We argue the answer is yes, under conditions we characterise precisely. We present Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes. Unlike prior work that grounds LLMs in pre-defined skill libraries or uses code generation for one-shot plan synthesis, AOR makes the full low-level motor control implementation the unit of LLM reasoning, enabling the agent to change not just what the robot does, but how it does it. The central claim is that interpretable code as the policy representation creates a qualitatively different kind of in-context learning from opaque neural policies: the agent can diagnose systematic failures and rewrite their causes. We validate this across three robosuite manipulation tasks and report promising results, with the agent achieving high success rates without demonstrations, reward engineering, or gradient updates.