MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

TL;DR

MemRL introduces a novel reinforcement learning approach on episodic memory that enables self-evolving AI agents to improve continuously during runtime without weight updates, effectively balancing stability and plasticity.

Contribution

It proposes MemRL, a non-parametric, two-phase retrieval method that filters noise and enhances strategy identification, advancing lifelong learning capabilities.

Findings

Outperforms state-of-the-art baselines on multiple benchmarks.

Effectively balances stability and plasticity in continuous learning.

Enables runtime self-improvement without weight updates.

Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL.