SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization

TL;DR

SSL introduces a tiered reward system inspired by the 'sweet spot' concept, guiding reinforcement learning agents more effectively by emphasizing incremental progress, leading to improved sample efficiency and robustness across diverse tasks.

Contribution

The paper proposes Sweet Spot Learning (SSL), a novel tiered reward framework that enhances reinforcement learning by promoting optimal solution regions and improving training efficiency.

Findings

SSL preserves optimal solution ordering.

SSL improves gradient signal-to-noise ratio.

SSL achieves up to 2.5X sample efficiency gains.

Abstract

Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for training intelligent agents. However, existing methods typically employ binary rewards that fail to capture quality differences among trajectories achieving identical outcomes, thereby overlooking potential diversity within the solution space. Inspired by the ``sweet spot'' concept in tennis-the racket's core region that produces optimal hitting effects, we introduce \textbf{S}weet \textbf{S}pot \textbf{L}earning (\textbf{SSL}), a novel framework that provides differentiated guidance for agent optimization. SSL follows a simple yet effective principle: progressively amplified, tiered rewards guide policies toward the sweet-spot region of the solution space. This principle naturally adapts across diverse tasks: visual perception tasks leverage distance-tiered modeling to reward proximity, while complex reasoning tasks reward incremental progress toward promising solutions. We theoretically demonstrate that SSL preserves optimal solution ordering and enhances the gradient signal-to-noise ratio, thereby fostering more directed optimization. Extensive experiments across GUI perception, short/long-term planning, and complex reasoning tasks show consistent improvements over strong baselines on 12 benchmarks, achieving up to 2.5X sample efficiency gains and effective cross-task transferability. Our work establishes SSL as a general principle for training capable and robust agents.