Learning to Augment Synthetic Images for Sim2Real Policy Transfer

TL;DR

This paper introduces an optimized augmentation strategy for synthetic images to improve sim2real transfer of manipulation policies, enabling effective policy learning without real data by using domain randomization and search-based augmentation.

Contribution

It presents a novel method to optimize synthetic image augmentation for better sim2real transfer, independent of specific policies, using object localization as a proxy for efficient search.

Findings

Significantly improved accuracy on three real robot manipulation tasks.

Augmentation strategy enables policy learning without real images.

Approach effectively reduces the domain gap between synthetic and real data.

Abstract

Vision and learning have made significant progress that could improve robotics policies for complex tasks and environments. Learning deep neural networks for image understanding, however, requires large amounts of domain-specific visual data. While collecting such data from real robots is possible, such an approach limits the scalability as learning policies typically requires thousands of trials. In this work we attempt to learn manipulation policies in simulated environments. Simulators enable scalability and provide access to the underlying world state during training. Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data. We follow recent work on domain randomization and augment synthetic images with sequences of random transformations. Our main contribution is to optimize the augmentation strategy for sim2real transfer and to enable domain-independent policy learning. We design an efficient search for depth image augmentations using object localization as a proxy task. Given the resulting sequence of random transformations, we use it to augment synthetic depth images during policy learning. Our augmentation strategy is policy-independent and enables policy learning with no real images. We demonstrate our approach to significantly improve accuracy on three manipulation tasks evaluated on a real robot.

Figures

Tables

Table 1. TABLE I : Cube prediction error (in cm) for synthetic and real depth images evaluated separately for each of eleven transformations considered in this paper. The errors are averaged over 200 pairs of images and cube positions.

Transformation	Error in sim	Error in real
Identity	0.63 $\pm$ 0.50	6.52 $\pm$ 5.04
Affine	0.59 $\pm$ 0.45	4.83 $\pm$ 4.42
Cutout	1.19 $\pm$ 0.87	1.86 $\pm$ 2.45
Invert	0.88 $\pm$ 0.60	4.63 $\pm$ 3.28
Posterize	0.66 $\pm$ 0.48	5.54 $\pm$ 4.59
Scale	0.67 $\pm$ 0.47	6.00 $\pm$ 4.37
Sharpness	0.83 $\pm$ 0.49	5.48 $\pm$ 3.84
WhiteNoise	0.68 $\pm$ 0.54	3.60 $\pm$ 2.33
SaltNoise	0.72 $\pm$ 0.50	2.42 $\pm$ 1.16
BoundaryNoise	0.88 $\pm$ 0.66	2.06 $\pm$ 1.17
EraseObject	0.64 $\pm$ 0.47	1.93 $\pm$ 1.01

Table 2. TABLE II : Cube prediction error (in cm) on synthetic and real depth images using different types of depth data augmentation. More augmentations increase the error for synthetic images and decrease the error for real images as expected from our optimization procedure. The errors are averaged over 200 pairs of images and cube positions.

	Augmentation	Error in sim	Error in real
i	None	0.63 $\pm$ 0.50	6.52 $\pm$ 5.04
ii	Random (8 operations)	6.56 $\pm$ 4.05	5.77 $\pm$ 3.12
iii	Handcrafted (4 operations)	0.99 $\pm$ 0.68	2.35 $\pm$ 1.36
iv	Learned (1 operation)	1.19 $\pm$ 0.87	1.86 $\pm$ 2.45
v	Learned (4 operations)	1.21 $\pm$ 0.78	1.17 $\pm$ 0.71
vi	Learned (8 operations)	1.31 $\pm$ 0.90	1.09 $\pm$ 0.73

Table 3. TABLE III : Success rates for control policies executed on a real robot (20 trials per experiment). Results are shown for three tasks and alternative depth image augmentations.

Augmentation	Pick	Stack	Cup Placing
None	3/20	1/20	0/20
Handcrafted (4 operations)	9/20	2/20	6/20
Learned (1 operation)	8/20	1/20	1/20
Learned (8 operations)	19/20	18/20	15/20

Equations

$$\lambda L_2\left([{\hat{\mathbf{v}}}_{\mathbf{t}},{\hat{\mathbf{\omega }}}_t],[{\mathbf{v}}_t,{\mathbf{\omega }}_t]\right)+(1-\lambda )L_{\text{CE}}\left(\widehat{g_t},g_t\right),$$

$$\sum _{k}E{L}_2\left(h\left(f(I_k^{\text{sim}})\right),p_k^{\text{sim}}\right).$$

$$e^{real}=\frac{1}{n}\sum _{k=1}^n{L}_2\left(h(I_k^{\text{real}}),p_k^{\text{real}}\right).$$

Full text

In this work we attempt to learn manipulation policies in simulated environments. Simulators enable scalability and provide access to the underlying world state during training. Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data. We follow recent work on domain randomization and augment synthetic images with sequences of random transformations. Our main contribution is to optimize the augmentation strategy for sim2real transfer and to enable domain-independent policy learning. We design an efficient search for depth image augmentations using object localization as a proxy task. Given the resulting sequence of random transformations, we use it to augment synthetic depth images during policy learning. Our augmentation strategy is policy-independent and enables policy learning with no real images. We demonstrate our approach to significantly improve accuracy on three manipulation tasks evaluated on a real robot.

11footnotetext: University Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France.22footnotetext: Inria, École normale supérieure, CNRS, PSL Research University, 75005 Paris, France.11footnotetext: Equal contribution.

I INTRODUCTION

Learning visuomotor control policies holds much potential for addressing complex robotics tasks in unstructured and dynamic environments. In particular, recent progress in computer vision and deep learning motivates new methods combining learning-based vision and control. Successful methods in computer vision share similar neural network architectures, but learn task-specific visual representations, e.g. for object detection, image segmentation or human pose estimation. Guided by this experience, one can assume that successful integration of vision and control will require learning of policy-specific visual representations for particular classes of robotics tasks.

Learning visual representations requires large amounts of training data. Previous work has addressed policy learning for simple tasks using real robots e.g., in [16, 22, 34]. Given the large number of required attempts (e.g. 800,000 grasps collected in [16]), learning with real robots might be difficult to scale to more complex tasks and environments. On the other hand, physics simulators and graphics engines provide an attractive alternative due to the simple parallelization and scaling to multiple environments as well as due to access to the underlying world state during training.

Learning in simulators, however, comes at the cost of the reality gap. The difficulty of synthesizing realistic interactions and visual appearance typically induces biases and results in low performance of learned policies in real scenes. Among several approaches to address this problem, recent work proposes domain randomization [30, 31] by augmenting synthetic data with random transformations such as random object shapes and textures. While demonstrating encouraging results for transferring simlator-trained policies to real environments (“sim2real” transfer), the optimality and generality of proposed transformations remains open.

In this work we follow the domain randomization approach and propose to learn transformations optimizing sim2real transfer. Given two domains, our method finds policy-independent sequences of random transformations that can be used to learn multiple tasks. While domain randomization can be applied to different stages of a simulator, our goal here is the efficient learning of visual representations for manipulation tasks. We therefore learn parameters of random transformations to bridge the domain gap between synthetic and real images. We here investigate the transfer of policies learned for depth images. However, our method should generalize to RGB inputs. Examples of our synthetic and real depth images used to train and test the “Cup placing” policy are illustrated in Fig. 1.

In more details, our method uses a proxy task of predicting object locations in a robot scene. We synthesize a large number of depth images with objects and train a CNN regressor estimating object locations after applying a given sequence of random transformations to synthetic images. We then score the parameters of current transformations by evaluating CNN location prediction on pre-recorded real images. Inspired by the recent success of AlphaGo [29] we adopt Monte-Carlo Tree Search (MCTS) [4] as an efficient search strategy for transformation parameters.

We evaluate the optimized sequences of random transformations by applying them to simulator-based policy learning. We demonstrate the successful transfer of such policies to real robot scenes while using no real images for policy training. Our method is shown to generalize to multiple policies. The overview of our approach is illustrated in Fig. 2. The code of the project is publicly available at the project website***http://pascal.inrialpes.fr/data2/sim2real.

II RELATED WORK

Robotics tasks have been addressed by various learning methods including imitation learning [8] and reinforcement learning [24]. Despite mastering complex simulated tasks such as Go [29] and Dota [19], the addressed robotics tasks remain rather simple [34, 24, 11]. This difference is mainly caused by the real world training cost. Learning a manipulation task typically requires a large amount of data [16, 22]. Thus, robot interaction time becomes much longer than in simulation [11] and expert guidance is non-trivial [34].

Learning control policies in simulation and transferring them to the real world is a potential solution to address these difficulties. However, the visual input in simulation is significantly different from the real world and therefore requires adaptation [27]. Recent attempts to bridge the gap between simulated and real images can be generally divided into two categories: domain adaptation [2] and domain randomization [30]. Domain adaptation methods either map both image spaces into a common one [14, 18] or map one into the other [15]. Domain randomization methods add noise to the synthetic images [21, 28], thus making the control policy robust to different textures and lighting. The second line of work is attractive due to its effectiveness and simplicity. Yet, it was so far only shown to work with RGB images. While depth images are well suited for many robotics tasks [17], it is not obvious what type of randomization should be used in the case of depth data. Here, we explore a learning based approach to select appropriate transformations and show that this allows us to close the gap between simulated and real visual data.

Domain randomization is also referred to as data augmentation in the context of image classification and object detection. Data augmentation is known to be an important tool for training deep neural networks and in most cases it is based on a manually designed set of simple transformations such as mirroring, cropping and color perturbations. In general, designing an effective data augmentation pipeline requires domain-specific knowledge [9]. Depth images might be augmented by adding random noise [12], noise patterns typical for real sensors [10] or by compensating missing information [33]. Learning to augment is a scalable and promising direction that has been explored for visual recognition in [20]. Recent attempts to automatically find the best augmentation functions propose to use Reinforcement Learning and require several hundreds of GPUs [6]. Given the prohibitive cost of executing thousands of policies in a real-robot training loop, we propose to optimize sequences of augmentations within a proxy task by predicting object locations in pre-recorded real images using the Monte Carlo Tree Search [4]. A related idea of learning rendering parameters of a simulator has been recently proposed for a different task of semantic image segmentation in [26].

III APPROACH

We describe the proposed method for learning depth image augmentations in Sections III-B and III-C. Our method builds on Behaviour Cloning (BC) policy learning [23, 25] which we overview in Section III-A.

III-A Behaviour cloning in simulation

Given a dataset $\mathcal{D}^{\textrm{expert}}=\{(o_{t},a_{t})\}$ of observation-action pairs along with the expert trajectories in simulation, we learn a function approximating the conditional distribution of the expert policy $\pi_{\textrm{expert}}(a_{t}|o_{t})$ controlling a robotic arm. Here, the observation is a sequence of the three last depth frames, $o_{t}=(I_{t-2},I_{t-1},I_{t})\in\mathcal{O}=\mathbb{R}^{H\times W\times 3}$. The action $a_{t}\in\mathcal{A}=\mathbb{R}^{7}$ is the robot command controlling the end-effector state. The action $a_{t}=(\mathbf{v}_{t},\bm{\omega}_{t},g_{t})$ is composed of the end-effector linear velocity $\mathbf{v}_{t}\in\mathbb{R}^{3}$, end-effector angular velocity $\bm{\omega}_{t}\in\mathbb{R}^{3}$ and the gripper openness state $g_{t}\in\{0,1\}$. We learn the deterministic control policy $\pi\mathrel{\mathop{\mathchar 58\relax}}\mathcal{O}\to\mathcal{A}$ approximating the expert policy $\pi_{\textrm{expert}}$. We define $\pi$ by a Convolutional Neural Network (CNN) parameterized by a set of weights $\theta$ and learned by minimizing the $L_{2}$ loss for the velocity controls $(\mathbf{v}_{t},\bm{\omega}_{t})$ and the cross-entropy loss $L_{\textrm{CE}}$ for the binary grasping signal $g_{t}$. Given the state-action pair $(s_{t},a_{t})$ and the network prediction $\pi_{\theta}(s_{t})=(\mathbf{\hat{v}}_{t},\bm{\hat{\omega}}_{t},\hat{g_{t}})$, we minimize the loss:

[TABLE]

where $\lambda\in[0,1]$ is a scaling factor of the cross-entropy loss which we experimentally set to 0.9.

III-B Sim2Real transfer

Given a stochastic augmentation function $f$, we train a CNN $h$ to predict the cube position on a simulation dataset \mathcal{D}^{\textrm{sim}}=\big{\{}(I^{\textrm{sim}}_{i},p^{\textrm{sim}}_{i})\big{\}}. Given an image $I^{\textrm{sim}}_{i}$, the function $f$ sequentially applies $N$ primitive transformations, each with a certain probability. This allows for bigger variability during the training. We minimize the $L_{2}$ loss between the cube position $p^{\textrm{sim}}_{i}$ and the network prediction given an augmented depth image $h\left(f(I^{\textrm{sim}}_{i})\right)$:

[TABLE]

We evaluate augmentation functions by computing the average error of network prediction on a pre-recorded real-world dataset, \mathcal{D}^{\textrm{real}}=\big{\{}(I^{\textrm{real}}_{i},p_{i}^{\textrm{real}})\big{\}} as

[TABLE]

The optimal augmentation function $f^{*}$ should result in a network $h$ with the smallest real-world error $e^{real}$. We assume that the same augmentation function will produce optimal control policies. We re-apply the learned stochastic function $f^{*}$ on individual frames of $\mathcal{D}^{\textrm{expert}}$ at every training epoch and learn $\pi^{\textrm{sim2real}}$. We use $\pi^{\textrm{sim2real}}$ to control the real robot using the same control actions as in the simulation, i.e., $a^{\textrm{real}}_{t}=(\mathbf{v}^{\textrm{real}}_{t},\bm{\omega}^{\textrm{real}}_{t},g^{\textrm{real}}_{t})=\pi^{\textrm{sim2real}}(I^{\textrm{real}}_{t})$, see Fig. 2.

III-C Augmentation space

We discretize the search space of augmentation functions by considering sequences of $N$ transformations from a predefined set. We then apply the selected sequence of transformations in a given order to each image. The predefined set of transformations consists of the depth-applicable standard transformations from PIL, a popular Python Image Library [1], as well as Cutout [7], white (uniform) noise and salt (bernoulli) noise. We also take advantage of segmentation masks provided by the simulator and define two object-aware transformations, i.e., boundary noise and object erasing (see Section IV-B for details). The identity (void) transformation is included in the set to enable the possibility of reducing $N$. The full set of our eleven transformations is listed in Table I. Each transformation is associated with a magnitude and a probability of its activation. The magnitude defines the transformation-specific parameter. For each transformation we define two possible magnitudes and three probabilities. With $N=8$ in our experiments, our search space roughly includes $(11\times 2\times 3)^{8}\approx 3.6*10^{14}$ augmentation functions. We reduce the search space by restricting each transformation, except identity, to occur only once in any augmentation sequence.

III-D Real robot control

To find an optimal sequence of augmentations, we use Monte Carlo Tree Search (MCTS) [4] which is a heuristic tree search algorithm with a trade-off between exploration and exploitation. Our search procedure is defined in Algorithm 1. The algorithm iteratively explores the Monte Carlo tree (lines 3-8) by sampling sequences of transformations (line 4), training a cube position prediction network on an augmented simulation dataset (line 5), evaluating the trained network on the real dataset (line 6) and backpropagating the evaluation error through the Monte Carlo tree (line 7). Once the smallest error on the real dataset stays constant for 500 iterations, we choose the best augmentation function according to MCTS (line 9) and train sim2real control policies using the simulation dataset of augmented expert trajectories on the tasks of interest (line 10). Once trained, the control policies can be directly applied in real robot scenes without finetuning. The sequence of eight transformations found by MCTS is illustrated in Fig. 3.

IV RESULTS

This section evaluates the transfer of robot control policies from simulation to real. First, we describe our tasks and the experimental setup in Section IV-A. We evaluate independently each of predefined basic transformations on the cube position prediction task in Section IV-B. In Section IV-C, we compare our approach of learning augmentation functions with baselines. Finally, we demonstrate the policy transfer to the real-world robotics tasks in Section IV-D.

IV-A Experimental setup

Our goal is to learn a policy to control a UR5 6-DoF robotic arm with a 3 finger Robotiq gripper for solving manipulation tasks in the real world. The policy takes as input 3 depth images $o_{t}\in\mathbb{R}^{H\times W\times 3}$ from the Kinect-1 camera positioned in front of the arm. We scale the values of depth images to the range $[0,1]$. The policy controls the robot with an action $a_{t}\in\mathbb{R}^{7}$. The control is performed at a frequency of 10 Hz. All the objects and the robotic gripper end-effector are initially allocated within the area of $60\times 60\,\text{cm}^{2}$ in front of the arm. The simulation environment is built with the pybullet physics simulator [5] and imitates the real world setup. We consider three manipulation tasks. Cube picking task: The goal of the task is to pick up a cube of size 4.7 cm and to lift it. In simulation, the cube size is randomized between 3 and 9 cm. Cubes stacking task: The goal of the task is to stack a cube of size 3.5 cm on top of a cube of size 4.7 cm. We randomize the sizes of cubes in simulation between 3 to 9 cm. Cup placing task: The goal of the task is to pick up a cup and to place it on a plate. In simulation, we randomly sample 43 plates from ModelNet [32] and 134 cups from ShapeNet [3]. We use three cups and three plates of different shapes in our real robot experiments.

IV-B Evaluation of individual transformations

Before learning complex augmentation functions, we independently evaluate each transformation from the set of transformations defined in Section III-C. In this section, we describe each transformation and the associated values of magnitude. Affine randomly translates and rotates the image in the range of $[-9,9]$ pixels and $[-5,5]\degree$, or alternatively by $[-16,16]$ pixels and $[-10,10]\degree$. Cutout [7] samples one or three random rectangles in the image and sets their values to a random constant in $[0,1]$. Invert function inverts each pixel value by applying the operation $x\mapsto 1-x$ and does not have any parameters. Posterize reduces the number of bits for each pixel value to be either 5 or 7. Scale randomly multiplies the image with a constant in one of the two ranges: $[0.95,1.05]$ or $[0.97,1.03]$. Sharpness increases the image sharpness either randomly in the range between $50\%$ and $100\%$ or by $100\%$. WhiteNoise adds uniform noise to each pixel with a magnitude of 0.04 or 0.08. SaltNoise sets each pixel value to 1 with the probability 0.01 or 0.03. BoundaryNoise uses the semantics mask of the simulator and removes patches of pixels located at the boundary between different objects. For BoundaryNoise, we remove either 2 or 4 pixels along the boundaries. EraseObject removes either the table or the walls behind the robot using the semantics segmentation mask. All the above transformations are associated with a probability in the set $\{33\%,66\%,100\%\}$. For instance, the probability $33\%$ means that the transformation is applied 1 out of 3 times and 2 our of 3 times it acts as the identity function.

As explained in Section III-B, we evaluate each augmentation function by computing the prediction error of the cube position in real depth images after training the position regressor on simulated data. We collect 2000 pairs of simulated depth images and cube positions for training and 200 real depth images for evaluation. To be robust to viewpoint changes in real scenes, each simulated scene is recorded from five random viewpoints. We randomize the camera viewpoint in simulation around the frontal viewpoint by sampling the camera yaw angle in $[-15,15]\degree$, pitch angle in $[15,30]\degree$, and distance to the robot base in $[1.35,1.50]$ m. We only require the viewpoint of the real dataset to be within the simulated viewpoints distribution. Moreover, we allow to move the camera between different real robot experiments. We treat each transformation in the given set as a separate augmentation function (sequence of length 1). We use each of them independently to augment the simulation dataset. Next, we train a CNN based on ResNet-18 architecture [13] to predict the cube position given a depth image. During the network training, we compute the prediction error (3) on two validation datasets, 200 simulated images and 200 real images. To evaluate each augmentation function more robustly, we always start the CNN training at the same initial position. For each transformation, we report the cube position prediction error in Table I. With no data augmentation, the trained network performs well in simulation (error of 0.63 cm) but works poorly on real images (error of 6.52 cm). Cutout transformation reduces the regression error by more than 4 cm and indicates that it should be potentially combined with other transformations.

IV-C Augmentation function learning

We compare the augmentation function learned by our approach to several baselines on the task of estimating the cube position in Table II. The baselines include training the network on synthetic images (i) without any data augmentation, (ii) with augmentation sequences composed of 8 random transformations (average over 10 random sequences), (iii) with a handcrafted augmentation sequence of four transformations built according to our initial intuition: Scale, WhiteNoise, EraseObjects and SaltNoise, (iv) with the best single transformation from Section IV-B and (v) with the learned augmentation composed of 4 transformations. To have a robust score and exclude outliers, we compute the median error over evaluations of 10 training epochs.

Baselines (i)-(iii) with no augmentation learning demonstrate worst results on the real dataset. Results for learned transformations (iv)-(vi) show that more transformations with different probabilities help to improve the domain transfer. Table II also demonstrates the trade-off between the performance in different domains: the better augmentation works on the real dataset, the worse it performs in the simulation. Effectively, the learned augmentation shifts the distribution of simulated images towards the distribution of real images. As a consequence, the network performs well on the real images that are close to the training set distribution and works worse on the original simulated images that lie outside of the training set distribution. The best augmentation sequence found by our method is illustrated in Fig. 3 and contains the following transformations: Cutout, EraseObject, WhiteNoise, EdgeNoise, Scale, SaltNoise, Posterize, Sharpness. Learning an augmentation function of length 8 takes approximately 12 hours on 16 GPUs. The vast majority of this time is used to train the position estimation network while MCTS path sampling, evaluation and MCTS backpropagation are computationally cheap. We iteratively repeat the training and evaluation routine until the error does not decrease for a sufficiently long time (500 iterations). Once the sim2real augmentation is found, it takes approximately an hour to train the BC control policy.

IV-D Real robot control

In this section we demonstrate that the data augmentation learned for a proxy task transfers to other robotic control tasks. We collect expert demonstrations where the full state of the system is known and an expert script can easily be generated at training time. We augment the simulated demonstrations with the learned data augmentation and train BC policies without any real images. Moreover, we show that our augmentation is not object specific and transfers to tasks with new object instances not present in the set of expert demonstrations. For each task, we compare the learned augmentation function with 3 baselines: no augmentation, handcrafted augmentation and best single transformation. Each evaluation consists of 20 trials with random initial configurations. The results are reported in Table III.

Cube picking task. Success rates for policies learned with different augmentation functions are strongly correlated with results for cube position estimation in Table II. The policy without augmentation has a success rate 3/20. Single transformation and handcrafted augmentation have 9/20 and 8/20 successful trials respectively. The sim2real policy learned with our method succeeds 19 out of 20 times.

Cube stacking task. Given a more difficult task where more precision is required, the baseline approaches perform poorly and achieve the success rate of only 2/20 for the handcrafted augmentation. We observe most of the failure cases due to imprecise grasping and stacking. We successfully tested the learned data augmentation function on cubes of varying sizes which indicates high control precision. Overall, our method was able to stack cubes in 18 runs out of 20.

Cup placing. Solving the Cup placing task requires both precision and the generalization to previously unseen object instances. The policies are trained over a distribution of 3D meshes and thus leverage the large dataset available in the simulation. All baselines fail to solve the Cup placing except for the handcrafted augmentation which succeeds 6 times out of 20. Our approach is able to solve the task with the success rate of 15/20 despite the presence of three different instances of cups and plates never seen during training. These results confirm our hypothesis that the augmentations learned for a proxy task of predicting the cube position, generalize to new objects and tasks.

V CONCLUSIONS

In this work, we introduce a method to learn augmentation functions for sim2real policy transfer. To evaluate the transfer, we propose a proxy task of object position estimation that requires only a small amount of real world data. Our evaluation of data augmentation shows significant improvement over the baselines. We also show that the performance on the proxy task strongly correlates with the final policy success rate. Our method does not require any real images for policy learning and can be applied to various manipulation tasks. We apply our approach to solve three real world tasks including the task of manipulating previously unseen objects.

Acknowledgements. This work was supported in part by the ERC grants ACTIVIA and ALLEGRO, the Louis Vuitton / ENS Chair on Artificial Intelligence and the DGA project DRAAF.