Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs

TL;DR

Prompted Policy Search (ProPS) introduces a novel reinforcement learning framework that integrates linguistic and numerical reasoning via large language models, enhancing sample efficiency and generalization across diverse tasks.

Contribution

ProPS uniquely centers LLMs in the policy optimization process, enabling direct policy updates through combined semantic and numerical feedback, advancing RL capabilities.

Findings

Outperforms baseline algorithms on 8 of 15 tasks

Incorporating domain knowledge improves learning efficiency

LLMs can perform in-context numerical optimization

Abstract

Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting its ability to leverage the rich semantic knowledge often available in real-world tasks. In contrast, humans learn efficiently by combining numerical feedback with language, prior knowledge, and common sense. We introduce Prompted Policy Search (ProPS), a novel RL method that unifies numerical and linguistic reasoning within a single framework. Unlike prior work that augment existing RL components with language, ProPS places a large language model (LLM) at the center of the policy optimization loop-directly proposing policy updates based on both reward feedback and natural language input. We show that LLMs can perform numerical optimization in-context, and that incorporating semantic signals, such as goals, domain knowledge, and strategy hints can lead to more informed exploration and sample-efficient learning. ProPS is evaluated across fifteen Gymnasium tasks, spanning classic control, Atari games, and MuJoCo environments, and compared to seven widely-adopted RL algorithms (e.g., PPO, SAC, TRPO). It outperforms all baselines on eight out of fifteen tasks and demonstrates substantial gains when provided with domain knowledge. These results highlight the potential of unifying semantics and numerics for transparent, generalizable, and human-aligned RL.