FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

TL;DR

FlashSAC is a novel off-policy reinforcement learning algorithm that achieves fast, stable, and efficient high-dimensional robot control by scaling models and data throughput while controlling error accumulation.

Contribution

It introduces FlashSAC, which reduces gradient updates and explicitly bounds norms to improve stability and efficiency in high-dimensional RL tasks.

Findings

Outperforms PPO and baselines on 60+ tasks in simulators.

Reduces training time from hours to minutes in sim-to-real humanoid locomotion.

Achieves superior performance and efficiency, especially in high-dimensional tasks.

Abstract

Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.