SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

TL;DR

SELAUR introduces a reinforcement learning framework for LLM agents that leverages intrinsic uncertainty signals to improve exploration, learning stability, and success rates in decision-making tasks.

Contribution

It is the first to incorporate token-level uncertainty metrics into reward design for LLM reinforcement learning, enhancing exploration and robustness.

Findings

Improves success rates on ALFWorld and WebShop benchmarks.

Uncertainty signals enhance exploration and robustness.

Ablation studies confirm the effectiveness of uncertainty integration.

Abstract

Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where effective reward design is essential for guiding learning. Although recent work explores various forms of reward shaping and step-level credit assignment, a key signal remains largely overlooked: the intrinsic uncertainty of LLMs. Uncertainty reflects model confidence, reveals where exploration is needed, and offers valuable learning cues even in failed trajectories. We introduce SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards, a reinforcement learning framework that incorporates uncertainty directly into the reward design. SELAUR integrates entropy-, least-confidence-, and margin-based metrics into a combined token-level uncertainty estimate, providing dense confidence-aligned supervision, and employs a failure-aware reward reshaping mechanism that injects these uncertainty signals into step- and trajectory-level rewards to improve exploration efficiency and learning stability. Experiments on two benchmarks, ALFWorld and WebShop, show that our method consistently improves success rates over strong baselines. Ablation studies further demonstrate how uncertainty signals enhance exploration and robustness.