Exploiting Policy Idling for Dexterous Manipulation

TL;DR

This paper introduces Pause-Induced Perturbations (PIP), a novel method that detects and perturbs idling states in dexterous manipulation policies, significantly improving robustness and success rates without extra supervision.

Contribution

The paper proposes PIP, a new exploration strategy that leverages idling detection to enhance policy learning and robustness in dexterous manipulation tasks.

Findings

PIP improves test-time performance in simulated tasks.

Learning from episodes with idling leads to better policy iteration.

Real-world insertion success rate increases by 15-35%.

Abstract

Learning-based methods for dexterous manipulation have made notable progress in recent years. However, learned policies often still lack reliability and exhibit limited robustness to important factors of variation. One failure pattern that can be observed across many settings is that policies idle, i.e. they cease to move beyond a small region of states when they reach certain states. This policy idling is often a reflection of the training data. For instance, it can occur when the data contains small actions in areas where the robot needs to perform high-precision motions, e.g., when preparing to grasp an object or object insertion. Prior works have tried to mitigate this phenomenon e.g. by filtering the training data or modifying the control frequency. However, these approaches can negatively impact policy performance in other ways. As an alternative, we investigate how to leverage the detectability of idling behavior to inform exploration and policy improvement. Our approach, Pause-Induced Perturbations (PIP), applies perturbations at detected idling states, thus helping it to escape problematic basins of attraction. On a range of challenging simulated dual-arm tasks, we find that this simple approach can already noticeably improve test-time performance, with no additional supervision or training. Furthermore, since the robot tends to idle at critical points in a movement, we also find that learning from the resulting episodes leads to better iterative policy improvement compared to prior approaches. Our perturbation strategy also leads to a 15-35% improvement in absolute success rate on a real-world insertion task that requires complex multi-finger manipulation.