SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

TL;DR

This paper introduces SEA-Eval, a new benchmark for evaluating self-evolving agents that can learn across tasks, formalizes their architecture, and highlights the importance of sequential convergence over success rate.

Contribution

It provides the first formal definition of Self-Evolving Agents, formalizes the Evolutionary Flywheel architecture, and develops SEA-Eval for comprehensive evaluation of evolutionary capabilities.

Findings

Token consumption varies up to 31.2× across frameworks with same success rates.

Sequential analysis reveals divergent evolutionary trajectories.

Success rate alone can create a capability illusion.

Abstract

Current LLM-based agents demonstrate strong performance in episodic task execution but remain constrained by static toolsets and episodic amnesia, failing to accumulate experience across task boundaries. This paper presents the first formal definition of the Self-Evolving Agent (SEA), formalizes the Evolutionary Flywheel as its minimal sufficient architecture, and introduces SEA-Eval -- the first benchmark designed specifically for evaluating SEAs. Grounded in Flywheel theory, SEA-Eval establishes $SR$ and $T$ as primary metrics and enables through sequential task stream design the independent quantification of evolutionary gain, evolutionary stability, and implicit alignment convergence. Empirical evaluation reveals that under identical success rates, token consumption differs by up to 31.2$\times$ across frameworks, with divergent evolutionary trajectories under sequential analysis -- demonstrating that success rate alone creates a capability illusion and that the sequential convergence of $T$ is the key criterion for distinguishing genuine evolution from pseudo-evolution.