Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning

TL;DR

OmniReset is a scalable reinforcement learning framework that uses systematic simulator resets to enable robust, long-horizon dexterous manipulation policies without task-specific engineering or demonstrations.

Contribution

The paper introduces OmniReset, a novel reset-based approach that simplifies exploration and scales to complex manipulation tasks using minimal human input.

Findings

OmniReset outperforms existing methods on long-horizon manipulation tasks.

Policies trained with OmniReset transfer effectively to real-world zero-shot.

OmniReset achieves broader behavioral coverage with minimal human-designed resets.

Abstract

Reinforcement learning in massively parallel physics simulations has driven major progress in sim-to-real robot learning. However, current approaches remain brittle and task-specific, relying on extensive per-task engineering to design rewards, curricula, and demonstrations. Even with this engineering, they often fail on long-horizon, contact-rich manipulation tasks and do not meaningfully scale with compute, as performance quickly saturates when training revisits the same narrow regions of state space. We introduce OmniReset, a simple and scalable framework that enables on-policy reinforcement learning to robustly solve a broad class of dexterous manipulation tasks using a single reward function, fixed algorithm hyperparameters, no curricula, and no human demonstrations. Our key insight is that long-horizon exploration can be dramatically simplified by using simulator resets to systematically expose the RL algorithm to the diverse set of robot-object interactions which underlie dexterous manipulation. OmniReset programmatically generates such resets with minimal human input, converting additional compute directly into broader behavioral coverage and continued performance gains. We show that OmniReset gracefully scales to long-horizon dexterous manipulation tasks beyond the capabilities of existing approaches and is able to learn robust policies over significantly wider ranges of initial conditions than baselines. Finally, we distill OmniReset into visuomotor policies which display robust retrying behavior and substantially higher success rates than baselines when transferred to the real world zero-shot. Project webpage: https://weirdlabuw.github.io/omnireset/