SpeedAug: Policy Acceleration via Tempo-Enriched Policy and RL Fine-Tuning

TL;DR

SpeedAug is a reinforcement learning framework that efficiently accelerates robotic policies by leveraging tempo-enriched pre-training, leading to faster execution and improved sample efficiency without sacrificing success rates.

Contribution

It introduces a novel tempo-enriched pre-training approach that enhances RL-based policy acceleration for robotic manipulation tasks.

Findings

Significantly improves sample efficiency of RL fine-tuning.

Maintains high success rates in robotic manipulation benchmarks.

Enables faster task execution without additional demonstrations.

Abstract

Recent advances in robotic policy learning have enabled complex manipulation in real-world environments, yet the execution speed of these policies often lags behind hardware capabilities due to the cost of collecting faster demonstrations. Existing works on policy acceleration reinterpret action sequence for unseen execution speed, thereby encountering distributional shifts from the original demonstrations. Reinforcement learning is a promising approach that adapts policies for faster execution without additional demonstration, but its unguided exploration is sample inefficient. We propose SpeedAug, an RL-based policy acceleration framework that efficiently adapts pre-trained policies for faster task execution. SpeedAug constructs behavior prior that encompasses diverse tempos of task execution by pre-training a policy on speed-augmented demonstrations. Empirical results on robotic manipulation benchmarks show that RL fine-tuning initialized from this tempo-enriched policy significantly improves the sample efficiency of existing RL and policy acceleration methods while maintaining high success rate.