RewardHarness: Self-Evolving Agentic Post-Training

TL;DR

RewardHarness introduces a self-evolving, agentic reward framework that improves image editing evaluation with minimal human data, surpassing existing models in accuracy and efficiency.

Contribution

It reframes reward modeling as context evolution, enabling iterative tool and skill development from limited preference demonstrations without large-scale annotations.

Findings

Achieves 47.4% accuracy on image-editing benchmarks using only 0.05% of preference data.

Surpasses GPT-5 by 5.3 points in evaluation accuracy.

Enhances reward modeling efficiency through self-evolving tool and skill libraries.

Abstract

Evaluating instruction-guided image edits requires rewards that reflect subtle human preferences, yet current reward models typically depend on large-scale preference annotation and additional model training. This creates a data-efficiency gap: humans can often infer the target evaluation criteria from only a few examples, while models are usually trained on hundreds of thousands of comparisons. We present RewardHarness, a self-evolving agentic reward framework that reframes reward modeling as context evolution rather than weight optimization. Instead of learning from large-scale annotations, RewardHarness aligns with human preferences by iteratively evolving a library of tools and skills from as few as 100 preference demonstrations. Given a source image, candidate edited images, and an editing instruction, an Orchestrator selects the most relevant subset of tools and skills from the maintained library, and a frozen Sub-Agent uses them to construct a reasoning chain that produces a preference judgment. By comparing predicted judgments with ground-truth preferences and analyzing successes and failures in the reasoning process, the Orchestrator automatically refines its library of tools and skills without additional human annotation. Using only 0.05% of the EditReward preference data, RewardHarness achieves 47.4% average accuracy on image-editing evaluation benchmarks, surpassing GPT-5 by 5.3 points. When used as a reward signal for GRPO fine-tuning, RL-tuned models achieve 3.52 on ImgEdit-Bench. Project page: https://rewardharness.com.