Sensory-Motor Control with Large Language Models via Iterative Policy Refinement

TL;DR

This paper introduces a novel iterative prompting method that enables large language models to generate and refine control policies for embodied agents, effectively integrating symbolic reasoning with sensory-motor data to solve control tasks.

Contribution

The paper presents a new approach for LLMs to control embodied agents by iteratively refining control policies through feedback, combining symbolic reasoning with sensory-motor data.

Findings

Effective on classic control tasks from Gymnasium and MuJoCo

Successfully identifies near-optimal solutions with compact models

Integrates symbolic reasoning with sensory-motor data

Abstract

We propose a method that enables large language models (LLMs) to control embodied agents through the generation of control policies that directly map continuous observation vectors to continuous action vectors. At the outset, the LLMs generate a control strategy based on a textual description of the agent, its environment, and the intended goal. This strategy is then iteratively refined through a learning process in which the LLMs are repeatedly prompted to improve the current strategy, using performance feedback and sensory-motor data collected during its evaluation. The method is validated on classic control tasks from the Gymnasium library and the inverted pendulum task from the MuJoCo library. The approach proves effective with relatively compact models such as GPT-oss:120b and Qwen2.5:72b. In most cases, it successfully identifies optimal or near-optimal solutions by integrating symbolic knowledge derived through reasoning with sub-symbolic sensory-motor data gathered as the agent interacts with its environment.