RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

TL;DR

RLAnything introduces a dynamic reinforcement learning framework that jointly optimizes environment, policy, and reward models through closed-loop feedback, significantly enhancing performance across various large language model and agentic tasks.

Contribution

The paper presents a novel RL framework that automatically adapts environment, policy, and reward models in a unified system, improving learning efficiency and task performance.

Findings

Achieves up to 18.7% performance improvement on benchmark tasks.

Joint optimization of models outperforms human-labeled reward signals.

Automatic environment adaptation enhances training effectiveness.

Abstract

We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL