Vision-based Robotic Arm Imitation by Human Gesture

TL;DR

This paper presents a vision-based method for robotic arm imitation using deep reinforcement learning, enabling a robot to learn complex tasks by observing human gestures through monocular camera input.

Contribution

It introduces a novel approach combining modified DDPG and visual imitation networks that learns from monocular camera frames without 3D environment reconstruction.

Findings

Robot can imitate human hand movements closely during training

Method requires only monocular camera input, no 3D modeling

Achieves effective visual imitation without explicit environment modeling

Abstract

One of the most efficient ways for a learning-based robotic arm to learn to process complex tasks as human, is to directly learn from observing how human complete those tasks, and then imitate. Our idea is based on success of Deep Q-Learning (DQN) algorithm according to reinforcement learning, and then extend to Deep Deterministic Policy Gradient (DDPG) algorithm. We developed a learning-based method, combining modified DDPG and visual imitation network. Our approach acquires frames only from a monocular camera, and no need to either construct a 3D environment or generate actual points. The result we expected during training, was that robot would be able to move as almost the same as how human hands did.

Figures

Equations

$${\nabla }_{{\theta }^{\mu }}J\approx \frac{1}{N}\sum _i{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_i,a=\mu (s_i)}{\nabla }_{{\theta }^{\mu }}\mu (s|{\theta }^{\mu }){|}_{s_i}$$

$${\mu }^{\prime }(s_t)=\mu (s_t|{\theta }^{\mu })+N$$

$${\mu }^{\prime }(s_t)=\mu (s_t|{\theta }^{\mu })+H_t$$

Full text

Vision-based Robotic Arm Imitation by Human Gesture

**Cheng Xuan, Zhiqiang Tang, Jinxin Xu

**Tsinghua University, University of Alberta

Beijing, People’s Republic of China, Edmonton, Canada

xc17@mails.tsinghua.edu.cn, ztang4@ualberta.ca, jinxin3@ualberta.ca 111Authors contributed equally

Abstract

• One of the most efficient ways for a learning-based robotic arm to learn to process complex tasks as human, is to directly learn from observing how human complete those tasks, and then imitate. Our idea is based on success of Deep Q-Learning (DQN) algorithm according to reinforcement learning, and then extend to Deep Deterministic Policy Gradient (DDPG) algorithm. We developed a learning-based method, combining modified DDPG and visual imitation network. Our approach acquires frames only from a monocular camera, and no need to either construct a 3D environment or generate actual points. The result we expected during training, was that robot would be able to move as almost the same as how human hands did.

1.   Introduction

Designing a robotic arm control system which aims at everyday tasks is one of the biggest challenges among current robot researches.

Recently, as Machine Learning being more and more popular, Deep Reinforcement Learning (DRL) have performed decently well dealing with some general and simple tasks, such as letting robotic arms learn to open doors with different kinds of handles after a lot of training (Yahya et all., 2016). However, the idea of letting robots work in vary situations and try to solve a same task in all those different cases is sort of intuitive. The main problem is that such idea might require several months’ training (Levine et all., 2016), and therefore it may not as practical as we expected in everyday tasks.

Our primary goal is to let the robotic arm itself learn from human, and to touch, push or even pick up items according to how exactly human do. This approach is able to significantly decrease the number of training steps by visually imitate human demonstrations. Further more, unlike other approaches of robot controlling with DDPG, which applied a random exploration noise while selecting an action, our algorithm allows robot to explore on purpose according to human movement, so that it will be more efficient in finding the best solution. A primary contributions we have done is that we modified DDPG algorithm, where we applied a Heuristic value to help to choose next actions, and we call that a Heuristic DDPG. This approach significantly reduce those unnecessary explorations accelerate our training efficiency, and since this approach is able to help robot to learn from human, our robot can perform in a human-like way, which makes it practical dealing with everyday tasks.

2.   Algorithm

Our method has two components. To generate heuristic values, we built a visual imitation neural network which used a deep convolutional neural network (CNN) to compare the similarity which is represented by our heuristic values, between human movement and robotic move trajectories. Moreover, we used a modified DDPG (Lillicarp et all., 2016) to continuously control the robot based on current frame from the camera and heuristic values acquired from imitation neural network.

To make the overall framework more clear, let’s go through this in following procedures. we firstly moved our hands from up to down and from left to right in front of the camera and recorded a video. Then, we moved our robot arm and let it process what human just did. Next, we built a convolutional neural network which took both robot and human hands’ frames of motion as input and locations of either the robotic arm or human hands in frames as output. After this, we were able to start to train the network and acquire heuristic values. After training imitation network, we can start testing our model on some simple tasks-pushing a button in front of the robot camera for example. Each frame gained from the camera will be feed into the imitation network and get estimated hands positions, and then we push every hand position into a stack which will turn into a sequence of data and form a trajectory. Finally, the robotic arm will start to move under the observation of camera and each frame will go through two parts. One is the imitation network to estimate current position of robot and then get a trajectory, the other is DDPG network to control robot based on its current state. Before actuating the robot, our system will compare the similarity of human hands and robot trajectories, and then generate a value of difference which is called heuristic action value. The actual movement for robot will be sum of heuristic value and DDPG action output, which actually will be able to reduce unnecessary exploration.

2.1 Data Acquisition and Preprocessing

Since our goal is to imitate human hands movement using our robotic arm, we firstly need to do a space mapping with robot and human. Unlike building a transformation matrix to describe the relative position of human hands and robot, we used the recorded images instead to acquire those positions which are represented by image coordinates.

We manually Tagged those images, and then tagged names by coordinate values of hands or robot. Finally we fed these images to the imitation net, and start to train our network. To increase the number of training datasets and robustness of the neural net, we did some image preprocessing-we increased the brightness and decreased the contrast, normalized standard deviation, and added salt and pepper noise on original images. Image translation, rotation, and flipping were not performed since the position information is important in this case.

2.2 Imitation Network

Our visual imitation network is a CNN based network, which has three convolution layers, two max pooling layers, and two fully connected layers. CNNs are commonly used in image classifications, and in our approach, it is used for position estimation of our robot and hands. The output is a type of one-hot encoding and has 25 values, we reshaped the 251 one dimensional output array into 55 two dimensional matrix to stand for 25 region in the image.

2.3 Heuristic DDPG net

DDPG algorithm is successfully used in continuously-especially high dimensional which in our case is $3^{6}$ different set of actions-control applications. Inspired from underlying Deep Q-Learning to the continuous action spaces. DDPG is an actor-critic(Sutton et all., 2015), model-free algorithm based on the deterministic policy gradient (Lillicarp et all., 2016). Since DDPG needs to do random exploration to find optimal actions, and however, continuous actions usually have large action spaces. As a result, training a DDPG network usually needs millions of steps, which may not be a problem for simulated tasks such as playing atari games, but will be problematic in real world application. For example, controlling a robotic arm which has seven or eight rollers. To conquer this challenge, we came up with an idea called Heuristic exploration.

A major problem of learning in continuous action spaces using reinforcement learning is useless exploration. The standard DDPG algorithm uses an exploration policy which adds some random value of noise while selecting actions,

[TABLE]

Where $\mu^{\prime}$ is exploration policy network and N is the noise value. Theoretically, DDPG is a random search algorithm and therefore unexceptionally inefficient. To improve the efficiency while exploring, we changed selection of actions based on,

[TABLE]

where $H_{t}$ is the heuristic value which is the result of comparing human and robot trajectories passing through our imitation network.

3.   References

Yahya, Ali, Li, Adrian, Kalakrishnan, Mrinal, Chebotar, Yevgen and Levine Sergey. Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search. arXiv:1610.00673.

Levine, Sergey, Pastor, Peter, Krizhevsky, Alex, Quillen, Deirdre. Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. arXiv:1603.02199

P. Lillicrap, Timothy, J. Hunt, Jonathan, Pritzel, Alexander et all. Continuous Control with Deep Reinforcement Learning. arXiv:1509.02971v5

S.Sutton, Richard, G.Barto, Andrew. Reinforcement Learning: An Introduction second edition. In Policy Approximation, p227-p230, 2015