Semi-Autonomous Teleoperation via Learning Non-Prehensile Manipulation Skills

TL;DR

This paper introduces a semi-autonomous teleoperation system that combines reinforcement learning and non-prehensile manipulation to improve pick-and-place tasks in cluttered environments, outperforming manual control.

Contribution

It presents a novel framework integrating learned non-prehensile manipulation strategies with teleoperation for efficient object rearrangement in cluttered scenes.

Findings

Outperforms manual keyboard control in task completion time

Successfully transfers policies from simulation to real-world

Handles varying numbers of objects effectively

Abstract

In this paper, we present a semi-autonomous teleoperation framework for a pick-and-place task using an RGB-D sensor. In particular, we assume that the target object is located in a cluttered environment where both prehensile grasping and non-prehensile manipulation are combined for efficient teleoperation. A trajectory-based reinforcement learning is utilized for learning the non-prehensile manipulation to rearrange the objects for enabling direct grasping. From the depth image of the cluttered environment and the location of the goal object, the learned policy can provide multiple options of non-prehensile manipulation to the human operator. We carefully design a reward function for the rearranging task where the policy is trained in a simulational environment. Then, the trained policy is transferred to a real-world and evaluated in a number of real-world experiments with the varying number of objects where we show that the proposed method outperforms manual keyboard control in terms of the time duration for the grasping.