Deep Reinforcement Learning for Robotic Manipulation under Distribution Shift with Bounded Extremum Seeking

TL;DR

This paper proposes a hybrid control method combining deep reinforcement learning with bounded extremum seeking to enhance robustness of robotic manipulation policies under distribution shifts.

Contribution

It introduces a novel hybrid controller that integrates DDPG policies with bounded extremum seeking for improved robustness in contact-rich tasks.

Findings

The hybrid controller maintains performance under out-of-distribution conditions.

Bounded ES enhances robustness to time variations and contact condition changes.

The approach is validated on pushing and pick-and-place tasks with varying environments.

Abstract

Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches.