Real-Time, Highly Accurate Robotic Grasp Detection using Fully Convolutional Neural Network with Rotation Ensemble Module

TL;DR

This paper introduces a rotation ensemble module for robotic grasp detection using CNNs, achieving high accuracy and real-time performance, especially for multiple objects and novel items.

Contribution

The paper presents a novel rotation ensemble module that enhances rotation invariance in grasp detection, outperforming existing methods in accuracy and speed.

Findings

Achieved up to 99.2% accuracy on Cornell dataset

Real-time grasp detection at 50 fps

93.8% success rate in robotic grasping of small objects

Abstract

Rotation invariance has been an important topic in computer vision tasks. Ideally, robot grasp detection should be rotation-invariant. However, rotation-invariance in robotic grasp detection has been only recently studied by using rotation anchor box that are often time-consuming and unreliable for multiple objects. In this paper, we propose a rotation ensemble module (REM) for robotic grasp detection using convolutions that rotates network weights. Our proposed REM was able to outperform current state-of-the-art methods by achieving up to 99.2% (image-wise), 98.6% (object-wise) accuracies on the Cornell dataset with real-time computation (50 frames per second). Our proposed method was also able to yield reliable grasps for multiple objects and up to 93.8% success rate for the real-time robotic grasping task with a 4-axis robot arm for small novel objects that was significantly higher than the baseline methods by 11-56%.

Figures

Tables

Table 1. TABLE I : Ablation studies on the Cornell dataset for anchor box of w 𝑤 w , h ℎ h with various ratios or one ratio and angle prediction methods with Reg, Cls, Rot.

Anchor Box	Angle Prediction	Image-wise		Object-wise
		25%	35%	25%	35%
N	Reg	91.0	86.5	88.7	85.6
1	Reg	91.8	87.7	89.2	86.3
N	Cls	97.2	93.1	96.1	93.1
1	Cls	97.3	94.1	96.6	92.9
1	Rot	98.3	94.4	96.6	93.6

Table 2. TABLE II : The ablation studies on the Cornell dataset for our REM with RC, RA and RL.

Angle	RC	RA	RL	Image-wise		Object-wise
				25%	35%	25%	35%
Cls	-	-	-	97.3	94.1	96.6	92.9
Cls	O	-	-	97.6	94.1	97.3	92.7
Cls	O	O	-	99.2	95.3	98.6	95.5
Cls	O	O	O	98.6	94.9	97.3	94.1
Reg	O	O	-	89.3	84.0	88.3	84.5
Rot	O	O	-	98.5	95.6	98.0	94.0

Table 3. TABLE III : Performance summary on Cornell dataset. Our proposed method yielded state-of-the-art prediction accuracy in both image-wise (Img) and object-wise (Obj) splits with real-time computation. The unit for performance is %.

Method	Angle	Type	Img	Obj	Speed
			25%	25%	(FPS)
Lenz [17], SAE	Cls	TSD	73.9	75.6	0.08
Redmon [23], AlexNet	Reg	OSD	88.0	87.1	13.2
Kumra [15], ResNet-50	Reg	TSD	89.2	88.9	16
Asif [2]	Reg	OSD	90.2	90.6	41
Guo [9]#a, ZFNet	Cls	TSD	93.2	82.8	-
Guo [9]#c, ZFNet	Cls	TSD	86.4	89.1	-
Chu [3], ResNet-50	Cls	TSD	96.0	96.1	8.3
Zhou [33]#b, ResNet-50	Rot	TSD	97.7	94.9	9.9
Zhou [33]#a, ResNet-101	Rot	TSD	97.7	96.6	8.5
Zhang [32], ResNet-101	Rot	TSD	93.6	93.5	25.2
Our REM, DarkNet-19	Cls	OSD	99.2	98.6	50

Table 4. TABLE IV : Performance summary of real robotic grasping tasks for 8 novel, small objects with 8 repetitions.

Object	Reg	Cls	Ours
toothbrush	5 / 8	8 / 8	8 / 8
candy	0 / 8	6 / 8	8 / 8
earphone cap	5 / 8	7 / 8	8 / 8
cable	3 / 8	6 / 8	7 / 8
styrofoam bowl	3 / 8	7 / 8	7 / 8
L-wrench	5 / 8	6 / 8	8 / 8
nipper	0 / 8	5 / 8	6 / 8
pencil	3 / 8	8 / 8	8 / 8
Average	3 / 8	6.6 / 8	7.5 / 8

Equations

$$(c_x,c_y)\in \{(c_x,c_y)|c_x,c_y\in \{0,1,\dots ,S-1\}\},$$

$$R(r)=\left[\begin{array}{ccc}\cos (r\pi /4)& -\sin (r\pi /4)& 0\\ \sin (r\pi /4)& \cos (r\pi /4)& 0\\ 0& 0& 1\end{array}\right]$$

$$d_l^i=g_l^i\star f,i=0,\dots ,3,l=1,\dots ,n_f,$$

$${\lambda }_{\mathrm{cd}}\left(\Vert m\odot (x-x^g){\Vert }_2+\Vert m\odot (y-y^g){\Vert }_2\right)+$$

$${\lambda }_{\mathrm{cd}}\left(\Vert m\odot (w-w^g){\Vert }_2+\Vert m\odot (h-h^g){\Vert }_2\right)+$$

$${\lambda }_{\mathrm{pr}}\Vert m\odot (z-z^g){\Vert }_2+{\lambda }_{\mathrm{ag}}\mathrm{AngLoss}({\theta }^g,\theta ;m)+$$

$$\frac{{\lambda }_{\mathrm{cd}}}{2}\left(\Vert m\odot (x_m-x^g){\Vert }_2+\Vert m\odot (y_m-y^g){\Vert }_2\right)+$$

$$\frac{{\lambda }_{\mathrm{cd}}}{2}\left(\Vert m\odot (w_m-w^g){\Vert }_2+\Vert m\odot (h_m-h^g){\Vert }_2\right)+$$

$$\frac{{\lambda }_{\mathrm{pr}}}{2}\Vert m\odot (z_m-z^g){\Vert }_2+\frac{{\lambda }_{\mathrm{ag}}}{2}\mathrm{CE}(m\odot {\theta }^g,m\odot {\theta }_m)$$

Full text

I INTRODUCTION

Robot grasping of novel objects has been investigated extensively, but it is still a challenging open problem in robotics. Humans instantly identify multiple grasps of novel objects (perception), plan how to pick them up (planning) and actually grasp it reliably (control). However, accurate robotic grasp detection, trajectory planning and reliable execution are quite challenging for robots. As the first step, detecting robotic grasps accurately and quickly from imaging sensors is an important task for successful robotic grasping.

Deep learning has been widely utilized for robotic grasp detection from a RGB-D camera and has achieved significant improvements over conventional methods. For the first time, Lenz et al. proposed deep learning classifier based robotic grasp detection methods that achieved up to 73.9% (image-wise) and 75.6% (object-wise) grasp detection accuracy on their in-house Cornell dataset [16, 17]. However, its computation time per image was still slow (13.5 sec per image) due to sliding windows. Redmon and Angelova proposed deep learning regressor based grasp detection methods that yielded up to 88.0% (image-wise) and 87.1% (object-wise) with remarkably fast computation time (76 ms per image) on the Cornell dataset [23]. Since then, there have been a lot of works proposing deep neural network (DNN) based methods to improve the performance in terms of detection accuracy and computation time. Fig. 1 summarizes the computation time (frame per second) vs. grasp detection accuracy on the Cornell dataset with object-wise split for some previous works (Redmon [23], Kumra [15], Asif [1], Chu [3], Zhou [33], Zhang [31]) and our proposed method. Note that recent works (except for our proposed method) using state-of-the-art DNNs such as [1, 3, 33, 31] seem to show trade-off between computation time and grasp detection accuracy. For example, Zhoua,b [33] were based on ResNet-101, ResNet-50 [11], respectively, that have the trade-off between network parameters vs. computation time. Note that prediction accuracy is generally related to real successful grasping and computation time is potentially related to real-time applications for fast moving objects or stand-alone applications with limited power.

Rotation invariance has been an important topic in computer vision tasks such as face detection [27], texture classification [8] and character recognition [13], to name a few. The importance of rotation invariant properties for computer vision methods still remains for recent DNN based approaches. In general, DNNs often require a lot more parameters with data augmentation with rotations to yield rotational-invariant outputs. Max pooling helps alleviating this issue, but since it is usually $2\times 2$ [12], it is only for images rotated with very small angles. Recently, there have been some works on rotation-invariant neural network such as rotating weights [4, 7], enlarged receptive field using dialed convolutional neural network (CNN) [29] or a pyramid pooling layer [10], rotation region proposals for recognizing arbitrarily placed texts [19] and polar transform network to extract rotation-invariant features [6].

Ideally, robot grasp detection should be rotation-invariant. Rotation angle prediction in robot grasp detection has been done by regression of continuous angle value [23], classification of discretized angles (e.g., $10^{\circ},20^{\circ},\ldots,170^{\circ}$) [9, 3] or rotation anchor box that is a hybrid method of regression and classification [32, 33, 31]. Previous works were not considering rotation-invariance or attempting rotation-invariant detection by rotating images or feature maps that were often time-consuming especially for multiple objects.

In this paper, we propose a rotation ensemble module (REM) for robotic grasp detection using convolutions that rotates network weights. This special structure allows the DNN to select rotation convolutions for each grid. Our proposed REM were evaluated for two different tasks: robotic grasp detection on the Cornell dataset [16, 17] and real robotic grasping tasks with novel objects that were not used during training. Our proposed REM was able to outperform state-of-the-art methods such as [33] by achieving up to 99.2% (image-wise), 98.6% (object-wise) accuracy on the Cornell dataset as shown in Fig. 1 with $5\times$ faster computation than [33]. Our proposed method was also able to yield up to 93.8% success rate for the real-time robotic grasping task with a 4-axis robot arm for novel objects and to yield reliable grasps for multiple objects unlike rotation anchor box.

II Related works

II-A Spatial, rotational invariance

Max pooling layers often alleviate the issue of spatial variance in CNN. To better achieve spatial-invariant image classification, Jaderberg et al. proposed spatial transformer network (STN), a method of image (or feature) transformation by learning (affine) transformation parameters so that it can help to improve the performance of inference operations of the following neural network layers [12]. Lin et al. proposed to use STN repeatedly with an inverse composite method by propagating warp parameters rather than images (or features) for improved performance [18]. Esteves et al. proposed a rotation-invariant network by replacing the grid generation of STN with a polar transform [6]. Input feature map (or image) was transformed into the polar coordinate with the origin that was determined by the center of mass. Cohen and Welling proposed a method to use group equivariant convolutions and pooling with weight flips and four rotations with $90^{\circ}$ stepsize [4]. Follmann et al. proposed to use rotation-invariant features that were created using rotational convolutions and pooling layers [7]. Marcos et al. proposed a network with a different set of weights for each local window instead of weight rotation [21].

II-B Object detection

Faster R-CNN was a method of using a region proposal network for generating region proposals to reduce computation time [26]. YOLO was faster but less accurate than the faster R-CNN by directly predicting {$x$, $y$, $w$, $h$, class} without using the region proposal network [24]. YOLO9000 stabilized the loss of YOLO by using anchor box inspired by region proposal network and yielded much faster object detection results than faster R-CNN while its accuracy was comparable [25]. For rotation-invariant object detection, Shi et al. investigated face detection using a progressive calibration network that predicted rotation by 180∘, 90∘ or an angle in [-45∘, 45∘] after sliding window [28]. Ma et al. used a rotation region proposal network to transform regions for classification using rotation region-of-interest (ROI) pooling [19]. Note that rotation angle was predicted using

1. rotation anchor box, 2) regression or 3) classification.

II-C Robotic grasp detection

Deep learning based robot grasp detection methods seem to belong one of the two types: two stage detector (TSD) or one stage detector (OSD). TSD consists of a region proposal network and a detector [9, 3, 32, 33, 31]. After extracting feature maps using proposals from the network in the first stage, objects are detected in the second stage. The region proposal network of TSD generally helps to improve accuracy, but is often time-consuming due to feature map extractions. OSD detects an object on each grid instead of generating region proposal to reduce computation time with decreased prediction accuracy [23].

Lenz et al. proposed a TSD model that classifies object graspability using a sparse auto-encode (SAE) with sliding windows for brute-force region proposals [17]. Redmon et al. developed a regression based OSD [23] using AlexNet [14]. Guo et al. applied ZFNet [30] based TSD to robot grasping and formulated angle prediction as classification [9]. Chu et al. further extended the TSD model of Guo [3] by incorporating recent ResNet [11]. Zhou et al. also used ResNet for TSD, but proposed rotation anchor box [33]. Zhang et al. extended the TSD method of Zhou [33] by additionally predicting objects using ROI [32]. DexNet 2.0 is also TSD that predicts grasp candidates from a depth image and then selects the best one by its classifier, GQ-CNN [20].

III Method

III-A Problem setup and reparametrization

The goal of the problem is to predict 5D representations for multiple objects from a color image where a 5D representation consists of location ($x$,$y$), rotation $\theta$, width $w$, and height $h$, as illustrated in Fig. 2. Multi-grasp detection often directly estimates 5D representation $\{x,y,\theta,w,h\}$ as well as its probability (confidence) of being a class (or being graspable) $z$ for each grid cell. In summary, the 5D representations with its probability are $\{x,y,\theta,w,h,z\}.$

For TSD, region proposal networks generate potential candidates for $\{x,y,w,h\}$ [3, 9, 33, 32] and rotation region proposal network yields possible arbitrary-oriented proposals $\{x,y,\theta,w,h\}$ [19]. Then, classification is performed for proposals to yield their graspable probabilities $z$. Rotation region proposal network classifies rotation anchor boxes with $30^{\circ}$ stepsize and then regresses angles.

For OSD, a set of $\{x,y,\theta,w,h,z\}$ is directly estimated [23]. Inspired by YOLO9000 [25], we propose to use the following reparametrization for 5D grasp representation and its probability for robotic grasp detection as $\{t^{x},t^{y},\theta,t^{w},t^{h},t^{z}\}$ where $x=\sigma(t^{x})+c_{x},y=\sigma(t^{y})+c_{y},$ $w=p_{w}\exp(t^{w}),h=p_{h}\exp(t^{h})$ and $z=\sigma(t^{z}).$ Note that $\sigma(\cdot)$ is a sigmoid function, $p_{h}$, $p_{w}$ are the predefined height and width of anchor box, respectively, and $c_{x},c_{y}$ are the top left corner of each grid cell. Therefore, a DNN directly estimates $\{t^{x},t^{y},\theta,t^{w},t^{h},t^{z}\}$ instead of $\{x,y,\theta,w,h,z\}$.

III-B Parameter descriptions of the proposed OSD method

For $S\times S$ grid cells, the following locations are defined

[TABLE]

which are the top left corner of each grid cell $(c_{x},c_{y})$. Thus, our proposed method estimates the $(x,y)$ offset from the top left corner of each grid cell. For a given $(c_{x},c_{y})$, the range of $(x,y)$ will be $c_{x}<x<c_{x}+1,\quad c_{y}<y<c_{y}+1$ due to the reparametrization using sigmoid functions.

We also adopt anchor box approach [25] to robotic grasp detection. Reparametrization changes regression for $w,h$ into regression & classification. Classification is performed to pick the best representation among all anchor box candidates that were generated using estimated $t^{w},t^{h}$ and the following $p_{w},p_{h}$ values: $\{(0.76,1.99),(0.76,3.2),(1.99,0.76),(1.99,\\ 1.99),(1.99,3.2),(3.2,3.2),(3.2,0.76)\}$ or $\{(1.99,1.99)\}$.

We investigated three prediction methods for rotation $\theta$. Firstly, a regressor predicts $\theta\in[0^{\circ},180^{\circ})$. Secondly, a classifier predicts $\theta\in\{0^{\circ},10^{\circ},\ldots,170^{\circ}\}$. Lastly, anchor box approach with regressor & classifier predicts both $\theta_{a}\in\{30^{\circ},90^{\circ},150^{\circ}\}$ and $\theta_{r}\in[-30^{\circ},30^{\circ}]$ to yield $\theta=\theta_{a}+\theta_{r}$.

Predicting detection (grasp) probability is crucial for multibox approaches such as MultiGrasp [23]. Conventional ground truth for detection probability was 1 (graspable) or 0 (not graspable) [23]. Inspired by [25], we proposed to use IOU (Intersection Over Union) as the ground truth detection probability as $z^{g}=|P\cap G|/|P\cup G|$ where $P$ is the predicted detection rectangle, $G$ is the ground truth detection rectangle, and $|\cdot|$ is the area of the rectangle.

III-C Rotation ensemble module (REM)

We propose a rotation ensemble module (REM) with rotation convolution and rotation activation to determine an ensemble weight associated with angle class probability for each grid. We added our REM to the latter part of a robot grasp detection network since it is often effective to put geometric transform related layers in the latter of the network such as deformable convolutions [5]. A typical location for REM in DNNs is illustrated in Fig. 3 (a).

Consider a typical scenario of convolution with input feature maps $f\in\mathbb{R}^{H\times W\times C}$ where $N=H\times W$ is the number of pixels and C is the number of channels. Let us denote $g_{l}\in\mathbb{R}^{K\times K\times C}$, $l=1,\ldots,n_{f}$ a convolution kernel where $K\times K$ is the spatial dimension of the kernel and there are $n_{f}$ number of kernels in each channel. Similar to the group convolutions [4], we propose $n_{r}$ rotations of the weights to obtain $n_{f}\cdot n_{r}$ rotated weights for each channel. Bilinear interpolations of four adjacent pixel values were used for generating rotated kernels. A rotation matrix is

[TABLE]

where $r$ is an index for rotations. Then, the rotated weights (or kernels) are $g_{l}^{i}=R(i)g_{l},i=0,\ldots,3,l=1,\ldots,n_{f}.$ Finally, the output of these convolutional layers with rotation operators for the input $f$ is

[TABLE]

where $\star$ is a convolution operator. This pipeline of operations is called “rotation convolution”. A typical kernel size is K=5.

Our REM contains rotation activation that aggregates all feature maps at different angles. Assume that an intermediate output for $\{t^{x},t^{y},\theta,t^{w},t^{h},t^{z}\}$ is available in REM, called $\{t^{x}_{m},t^{y}_{m},\theta_{m},t^{w}_{m},t^{h}_{m},t^{z}_{m}\}$. Note that ${\theta}_{m}^{i}\in\mathbb{R}^{H\times W}$ where $i=0,\pi/4,2\pi/4,3\pi/4$. For each angle, activations will be generated and all of them must be aggregated to yield one final feature map $\hat{d}_{l}=\sum^{4}_{i=1}d^{i}_{l}\odot\theta_{m}^{i}/4.$ where $\odot$ is Hadamard product. Thus, our proposed method utilizes class probability (probability to grasp) to selectively aggregate activations along with the weight of angle classification.

In the REM, the intermediate output is partially used for rotation activation, it still contains valuable, compressed information about the final output - it could be a good initial bounding box. Thus, we designed our REM to decompress, concatenate it at the end of REM as illustrated in Fig. 3 (b). This pipeline delivers valuable information about $\{t^{x}_{m},t^{y}_{m},\theta_{m},t^{w}_{m},t^{h}_{m},t^{z}_{m}\}$ indirectly to the final layer and this structure seemed to decrease probability errors.

III-D Loss function for REM-equipped DNN

We re-designed the loss function for training robotic grasp detection DNNs to emphasize this additional REM. The output of DNN $(t^{x},t^{y},\theta,t^{w},t^{h},t^{z})$ and the intermediate output of the REM $\{t^{x}_{m},t^{y}_{m},\theta_{m},t^{w}_{m},t^{h}_{m},t^{z}_{m}\}$ should be converted into $(x,y,\theta,w,h,z)$ and $\{x_{m},y_{m},\theta_{m},w_{m},h_{m},z_{m}\}$, respectively. Then, using the ground truth $(x^{g},y^{g},\theta^{g},w^{g},h^{g},z^{g})$, the loss function is defined as

[TABLE]

where $m$ is a mask vector with 1 (ground truth for that grid) or 0 (no ground truth for that grid), $\|\cdot\|_{2}$ is $l_{2}$ norm, CE is cross entropy, and AngLoss is one of these functions: CE for classification on $\theta$, $l_{2}$ norm for regression or rotation anchor box on $\theta$. We chose $\lambda_{\mathrm{cd}}=\lambda_{\mathrm{ag}}=1$ and $\lambda_{\mathrm{pr}}=5$.

IV Simulations and Experiments

We evaluated our proposed REM methods on the Cornell robotic grasp dataset  [16, 17] and on real robot grasping tasks with novel objects. The effectiveness of our REM was demonstrated in prediction accuracy, computation time and grasping success rate. Our proposed methods were compared with previous methods such as [17, 23, 9, 3, 33, 32] based on literature for widely used Cornell dataset as well as our in-house implementations of some previous works.

IV-A Implementation details

It is challenging to fairly compare a robot grasp detection method with other previous works such as [17, 23, 9, 3, 33, 32]. Due to the Cornell dataset, most works were able to compare their results with those of previous methods that were reported in literature. Considering fast advances of computing power and DNN techniques, it is often not clear how much the proposed scheme or method actually contributed to the increase of performance.

In this paper, we did not only compare our REM methods with previous works on the Cornell dataset through literature, but also implemented the core angle prediction schemes of other previous works with modern DNNs: regression (Reg) that Redmon et al. proposed [23], classification (Cls) that Guo et al. proposed [9] and rotation anchor box (Rot) that Zhou et al. proposed [33]. While Redmon [23], Guo [9] and Zhou [33] used AlexNet [14], ZFNet [30] and ResNet [11], respectively, our in-house implementations, Reg, Cls and Rot, all used DarkNet-19 [22]. While Guo and Zhou were based on faster R-CNN (TSD) [26], our implementations were based on YOLO9000 (OSD) [25].

We performed ablation studies for our REM so that it becomes clear which part will affect the performance of rotated grasp detection most significantly. We placed our proposed REM at the 6th layers from the end of the detection network. We also performed simulations with rotation activation using angle and probability. For multiple robotic grasps detection, boxes were plotted when probabilities were 0.25 or higher.

All algorithms were tested on the platform with GPU (NVIDIA 1080Ti), CPU (Intel i7-7700K 4.20GHz) and 32GB memory. Our REM methods and other in-house DNNs such as Ref, Cls and Rot were implemented with PyTorch.

IV-B Benchmark dataset and novel objects

The Cornell robot grasp detection dataset [16, 17] consists of $885$ images (RGB color and depth) of 240 different objects as shown in Fig. 4(a) with ground truth labels of a few graspable rectangles and a few non-graspable rectangles. We used RG-D information without B channel just like the work of Redmon [23]. An image was cropped to yield a $360\times 360$ image and five-fold cross validation was performed. Then, mean prediction accuracy was reported for image-wise and object-wise splits. Image-wise split divides the Cornell dataset into training and testing data with 4:1 ratio randomly without considering the same or different objects. Object-wise is a way of splitting training and testing data with 4: 1 ratio such that both data do not contain the same object. We followed other previous works for accuracy metrics [17, 23, 15]. Successful grasp detection is defined as follows: if IOU is larger than a certain threshold ($e.g.$, 0.25, 0.3 or 0.35) and the difference between the output orientation $\theta$ and the ground truth orientation $\theta^{g}$ is less than 30∘ (Jaccard index), then it is considered as a successful grasp detection.

We also performed real grasping tasks with our REM methods on 8 novel objects as shown in Fig. 4(b) (toothbrush, candy, earphone cap, cable, styrofoam bowl, L-wrench, nipper, pencil). Our proposed methods were applied to a small 4-axis robot arm (Dobot Magician, China) and a RGB-D camera (Intel RealSense D435, USA) that has a field-of-view of the robot and its workspace from the top. If a robot can pick and place an object, it is counted as success. Our robot experiment setup is illustrated in Fig. 5.

IV-C Results for in-house implementations of previous works

Table I shows the results of ablation studies for our in-house implementations on the Cornell dataset for anchor box with $w$ and $h$ with various ratios (N) vs. one ratio of 1:1 (1) and angle prediction methods: regression (Reg) vs. classification (Cls) vs. rotation anchor box (Rot). The results show that using a 1:1 ratio (1) yields better accuracy than using a variety of anchor boxes (N). For angle prediction methods, rotation anchor box yielded the best performance while regression yielded the lowest that was consistent with the literature. Thus, our in-house implementations seem to yield better performance in accuracy than the original previous works possibly due to modern DNNs in our implementations: Reg - Redmon et al. [23], Cls - Guo et al. [9] and Rot - Zhou et al. [33].

Fig. 6 shows the results of different angle prediction methods at IOU 25% over epoch. We observed that Rot yielded slowly increased accuracy over epochs than Cls initially and Reg yielded overall slow increase in accuracy over epochs. These slow initial convergences of Reg and Rot may not be desirable for re-training on additional data.

IV-D Results for our proposed REM on the Cornell dataset

Table II shows the results of the ablation studies for our proposed REM with different components such as rotation convolution (RC) and rotation activation (RA). RA can be obtained by using rotation activation loss (RL) as show in Fig. 3. We observed that RC itself did not improve the performance while RC & RA significantly improved the accuracy. Comparable performance was observed when using RC & RA with Rot, but substantially low performance was achieved with Reg.

Table III summarizes all evaluation results on the Cornell robotic grasp dataset for previous works and our proposed methods. Our proposed method yielded state-of-the-art performance, up to 99.2% prediction accuracy for image-wise split and up to 98.6% for object-wise split, respectively, over reported accuracies of the previous works that are listed in the Table. Our proposed methods yielded these state-of-the-art performances with real-time computation at 50 frame per second (FPS). Note that AlexNet, DarkNet-19, ResNet-50, ResNet-101 require 61.1, 20.8, 25.6 and 44.5 MB parameters, respectively. Thus, our REM method achieved state-of-the-art results with relatively small size of DNN (20.8MB) compared to other recent works using large DNNs such as ResNet-101 (44.5MB).

Fig. 7 illustrates grasp detection results on the Cornell dataset. Our proposed Cls+REM yielded grasp candidates that were close to the ground truth compared to other previous methods such as Reg and Cls.

IV-E Results for real grasping tasks on novel objects

We applied all grasp detection methods that were trained on the Cornell set to real grasping tasks with novel (multiple) objects without re-training. Fig. 8 illustrates our robot grasp experiment with novel objects including nipper using our algorithm implementations. Multi-object multi-grasp detection results on novel objects are reported in Fig. 9 for Reg, Cls, Rot and our Cls+REM methods, respectively. Both Cls and our Cls+REM generated good grasp candidates over Reg an Rot. Our REM seems to detect reliable grasps and angles (e.g, pen, L-wrench) over Rot. Real grasping task results with our 4-axis robot arm is tabulated in Table IV. Possibly due to reliable angle detections, our proposed Cls+REM yielded 93.8% grasping success rate, that is 11% higher than Cls. We did not perform real grasping with Rot, a modern version of Zhou [33], due to unreliable angle predictions for multiple objects. However, the advantage of our Cls+REM seems clear over Rot for detection accuracies, fast computation and reliable angle predictions for multi-objects.

V CONCLUSION

We propose the REM for robotic grasp detection that was able to outperform state-of-the-art methods by achieving up to 99.2% (image-wise), 98.6% (object-wise) accuracies on the Cornell dataset with fast computation (50 FPS) and reliable grasps for multi-objects. Our proposed method was able to yield up to 93.8% success rate for the real-time robotic grasping task with a 4-axis robot arm for small novel objects that was higher than the baseline methods by 11-56%.

Acknowledgments

This work was supported partly by the Technology Innovation Program or Industrial Strategic Technology Development Program (10077533, Development of robotic manipulation algorithm for grasping/assembling with the machine learning using visual and tactile sensing information) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea) and partly by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number : HI18C0316).