Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning

TL;DR

This paper introduces Progressive-Resolution Policy Distillation, a method that efficiently transfers policies from coarse to fine simulations in reinforcement learning for autonomous excavation, significantly reducing sampling time while maintaining success rates.

Contribution

The paper proposes a novel progressive policy transfer framework that leverages multi-resolution simulations to improve efficiency in RL-based autonomous excavation.

Findings

Reduced sampling time to less than 1/7 of traditional methods

Maintained high success rates comparable to fine-resolution training

Validated effectiveness in both simulated and real-world environments

Abstract

In earthwork and construction, excavators often encounter large rocks mixed with various soil conditions, requiring skilled operators. This paper presents a framework for achieving autonomous excavation using reinforcement learning (RL) through a rock excavation simulator. In the simulation, resolution can be defined by the particle size/number in the whole soil space. Fine-resolution simulations closely mimic real-world behavior but demand significant calculation time and challenging sample collection, while coarse-resolution simulations enable faster sample collection but deviate from real-world behavior. To combine the advantages of both resolutions, we explore using policies developed in coarse-resolution simulations for pre-training in fine-resolution simulations. To this end, we propose a novel policy learning framework called Progressive-Resolution Policy Distillation (PRPD), which progressively transfers policies through some middle-resolution simulations with conservative policy transfer to avoid domain gaps that could lead to policy transfer failure. Validation in a rock excavation simulator and nine real-world rock environments demonstrated that PRPD reduced sampling time to less than 1/7 while maintaining task success rates comparable to those achieved through policy learning in a fine-resolution simulation. Additional videos and supplementary results are available on our project page: https://yuki-kadokawa.github.io/prpd/