Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies

TL;DR

This paper introduces MLES, a novel method combining multimodal large language models with evolutionary search to discover transparent, human-aligned control policies that perform comparably to traditional reinforcement learning methods.

Contribution

It presents a new approach for programmatic control policy discovery using multimodal LLMs and evolutionary search, enabling transparent, adaptable policies without relying on predefined languages.

Findings

MLES achieves performance comparable to PPO in control tasks.

MLES produces transparent, human-readable control policies.

The method is scalable and facilitates knowledge transfer across tasks.

Abstract

Deep reinforcement learning has achieved impressive success in control tasks. However, its policies, represented as opaque neural networks, are often difficult for humans to understand, verify, and debug, which undermines trust and hinders real-world deployment. This work addresses this challenge by introducing a novel approach for programmatic control policy discovery, called Multimodal Large Language Model-assisted Evolutionary Search (MLES). MLES utilizes multimodal large language models as programmatic policy generators, combining them with evolutionary search to automate policy generation. It integrates visual feedback-driven behavior analysis within the policy generation process to identify failure patterns and guide targeted improvements, thereby enhancing policy discovery efficiency and producing adaptable, human-aligned policies. Experimental results demonstrate that MLES achieves performance comparable to Proximal Policy Optimization (PPO) across two standard control tasks while providing transparent control logic and traceable design processes. This approach also overcomes the limitations of predefined domain-specific languages, facilitates knowledge transfer and reuse, and is scalable across various tasks, showing promise as a new paradigm for developing transparent and verifiable control policies. Code is publicly available at https://github.com/QingL2000/MLES.