From Imitation to Refinement -- Residual RL for Precise Assembly

TL;DR

This paper introduces ResiP, a method combining Behavior Cloning with reinforcement learning residuals to improve the reliability and precision of robotic assembly tasks, overcoming distribution shift and open-loop limitations.

Contribution

ResiP is a novel approach that augments chunked BC models with RL-based residual policies for closed-loop correction, enhancing precision and reliability in robotic assembly.

Findings

ResiP improves task success rates in precise assembly.

ResiP effectively addresses distribution shift issues.

ResiP maintains BC's ease of teaching and long-horizon capabilities.

Abstract

Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks. However, we find that the ease of teaching comes at the cost of unreliable performance that saturates with increasing data for tasks requiring precision. The performance saturation can be attributed to two critical factors: (a) distribution shift resulting from the use of offline data and (b) the lack of closed-loop corrective control caused by action chucking (predicting a set of future actions executed open-loop) critical for BC performance. Our key insight is that by predicting action chunks, BC policies function more like trajectory "planners" than closed-loop controllers necessary for reliable execution. To address these challenges, we devise a simple yet effective method, ResiP (Residual for Precise Manipulation), that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities. ResiP augments a frozen, chunked BC model with a fully closed-loop residual policy trained with reinforcement learning (RL) that addresses distribution shifts and introduces closed-loop corrections over open-loop execution of action chunks predicted by the BC trajectory planner. Videos, code, and data: https://residual-assembly.github.io.