DenseAttentionSeg: Segment Hands from Interacted Objects Using Depth Input

TL;DR

DenseAttentionSeg is a real-time deep neural network that segments hands and objects from depth data using a dense attention mechanism, contour loss, and a new large-scale dataset, outperforming existing methods.

Contribution

The paper introduces DenseAttentionSeg with a novel dense attention mechanism, contour loss, and the InterSegHands dataset for improved interaction segmentation.

Findings

Outperforms state-of-the-art segmentation methods

Achieves real-time processing speeds

Provides a large-scale dataset for hand-object interactions

Abstract

We propose a real-time DNN-based technique to segment hand and object of interacting motions from depth inputs. Our model is called DenseAttentionSeg, which contains a dense attention mechanism to fuse information in different scales and improves the results quality with skip-connections. Besides, we introduce a contour loss in model training, which helps to generate accurate hand and object boundaries. Finally, we propose and release our InterSegHands dataset, a fine-scale hand segmentation dataset containing about 52k depth maps of hand-object interactions. Our experiments evaluate the effectiveness of our techniques and datasets, and indicate that our method outperforms the current state-of-the-art deep segmentation methods on interaction segmentation.

Tables

Table 1. Table 1: Statistics of several hand segmentation datasets.

Dataset	Type	Annotation	# Frames	Hand Resolution	Object Interaction
EYTH[13]	color	manual	1,290	Uncertain	Partial
NYU[1]	depth	automatic	6,736	Middle	No
HOI[2]	depth	-	27,525	Low	Yes
InterSegHands(Ours)	depth	automatic	51,912	High	Yes

Table 2. Table 2: Quantitative results on InterSegHands validation set. The mean IOU is per class mean intersection over union rate for hand and object classes. The time is the inference time of batch size 1 on GTX Titan Xp with an input size of 640 × 480 640 480 640\times 480 . CL : contour loss finetuning.

	mean IOU	pixel acc.	mean acc.	mean IU	f.w. IU	time
FCN-32s[3]	87.60	99.28	94.12	91.69	98.6	35.4ms
FCN-16s[3]	87.59	99.39	94.48	91.67	98.83	35.6ms
FCN-8s[3]	91.58	99.62	96.32	94.38	99.26	33.5ms
Large Kernel Matters (k=9)[19]	92.55	99.69	96.95	95.10	99.40	49.5ms
RefineNet[18]	92.53	99.64	96.74	94.98	99.30	61.9ms
DeepLabv3+ (Resnet-50, OS-16)[20]	92.76	99.64	96.99	95.16	99.31	47.8ms
DenseAttentionSeg	93.65	99.74	97.48	95.77	99.50	53.7ms
DenseAttentionSeg w/ CL	93.94	99.75	97.67	95.99	99.51	53.7ms

Table 3. Table 3: Evaluation on dense attention mechanism which increases the value of mean IOU from our baseline model.

	mean IOU	time
Deeplabv3+ (baseline)	92.76	47.8ms
Ours w/ attention	92.81	51.1ms
Ours wo/ attention	93.65	53.7ms

Equations

$$M_o=\bigcup \{s\in \mathcal{T}(M_i)|N(s)>\theta ,\frac{(\partial N(s){)}^2}{N(s)}<\varphi \}$$

$${\mathcal{L}}_{softmax}=-\sum _i{y}_i\log \frac{e^{x_i}}{{\sum }_j{e}^{x_j}}$$

$$\mathcal{S}(M)=\sum _{l=1}^{n-1}\sqrt{(K_h\ast M_l{)}^2+(K_v\ast M_l{)}^2}$$

$$\mathcal{B}(M)=G\ast M$$

$${\mathcal{L}}_{contour}=\frac{1}{N}\sum (\mathcal{B}(\mathcal{S}(M_{labels}))-\mathcal{B}(\mathcal{S}(M_{logits})){)}^2$$

$${\mathcal{L}}_{finetuning}=\alpha {\mathcal{L}}_{softmax}+\beta {\mathcal{L}}_{contour}$$

Full text

1 Introduction

Hand is an important part of our body that makes human be able to construct, manipulate tools and express ideas. Many researches, such as pose recognition, 3D reconstruction, human-computer interaction, AR/VR, etc., have paid a lot of attentions on human hands. Among almost all those works, it is important to detect and segment hands from their interacting objects and backgrounds, and hand-object segmentation can also be applied to many other high-level tasks.

Many existing works consider RGB images as input. However, RGB images are easily affected by the hand skin, the color of interacting objects and the illumination condition, which add inherent difficulties for hand-related applications. In this paper, we consider using depth images which are more robust to these factors, and become easier to be obtained as more and more consumer depth sensors are available even on mobile devices. And we propose a depth-based hand-object segmentation dataset to promote researches on this topic.

Existing hand segmentation datasets on depth either lack object interaction [1] or are almost in large scenes with low-resolution hands and indistinct boundaries [2]. A large dataset with high-resolution hand-object interaction and delicate pixel-level labels is eagerly required to meet the demand for network training in the deep learning era. So we propose InterSegHands to match the requirement and accompanying an auto-annotation method which is used to construct the current dataset and can be used to easily enlarge the dataset in the future. We will release the dataset and the auto-annotation method to promote researches in this area.

Nowadays, deep learning methods have been adopted in general semantic segmentation successfully [3, 4, 5]. In this work, we evaluate some famous deep segmentation models. Furthermore, we propose DenseAttentionSeg, which contains a dense attention structure to fuse the information of different scales and guide the skip-connections. Attention mechanism has been proven effective in CV and NLP, which often computes the attention weights of the encoder in seq-to-seq learning or different regions of the input feature map. Recently, similar ideas are also tried in general semantic segmentation [6, 7], but they are limited to high-level guidance. Nevertheless, we build an attention mechanism on different feature scales which not only convey coarser semantic but also finer shape and edge information, which we call it dense attention. In addition, we propose a new contour loss which emphasizes the object contours and promotes the final results a lot, which is particularly designed for this fine-scale hand-object segmentation job and will not decrease the inference speed.

The whole pipeline is shown in Figure 1. Our contributions can be summarized as follows: First, we introduce a large fine-scale dataset for hand segmentation with object interaction using depth maps, together with an auto-annotation method; Second, we propose an encoder-decoder architecture with a dense attention mechanism for this hand segmentation task. As far as we know, this is the first depth-based reconstruction-level segmentation work for hand-object interaction. It outperforms the state-of-the-art semantic segmentation models on our dataset; Third, we present a contour refinement strategy that significantly improves the accuracy of our segmentation result.

2 Related Work

As a basic task in hand research, hand segmentation contributes to many hand applications, such as pose estimation [8, 9], gesture recognition [10, 11] and so on.

Color and depth: As for RGB-based methods, Bambach et al. segmented hands egocentrically in many activities via the combination of traditional and neural methods [12]. Urooj et al. explored egocentric hand segmentation in social activities [13]. There are also some hand works (including hand tracking and gesture recognition) based on depth camera [14, 11]. Some methods used random decision forest(RDF) [1, 9]. Kang et al. adopted a simple segmentation method by using a black wristband in their hand tracking work [15]. They also studied the problem of hand segmentation recently [2].

Hand and object interaction: Though the most common situation of hands in our daily life is to interact with all kinds of objects, it has not been fully researched. Kang et al. proposed a two-stage RDF to segment hand and object on depth [2]. Their solution runs fast while with a fair accuracy. And most of their dataset samples lack closeup information of hand and object details. Another relevant work is Urooj et al.’s [13], in which they used a high-performance network to segment hand. But they adopted RGB images as input.

Datasets: Bambach et al. introduced EgoHands for hand tasks including segmentation based on RGB images [12], including 4.8k frames in total. Zimmermann and Brox recently proposed a larger dataset for color images, which has about 44k samples [16]. But it is synthetic and difficult to apply in the real world. As for the depth based dataset, Tompson et al. released NYU Hand Pose Dataset, which contains about 6k depth frames but lacks object interaction [1]. Kang et al. collected a total number of 27k depth images for segmentation, but most of their dataset samples do not show the fine structure of hands and lost the object information to some degree [2].

Network based semantic segmentation: Long et al. first proposed fully convolutional neural networks [3]. Ronneberger et al. designed an encoder-decoder architecture that was widely used in the following years [4]. Other methods include PSPNet [17], RefineNet [18], Large Kernel Matters [19], Deeplab [5], and so on. Though existing network-based models work well, they were built for general segmentation tasks and may not all act the same for depth-based hand-object interaction segmentation task where the classes are limited and the hand and object are closely related.

3 InterSegHands Dataset

Our database contains depth images of various hand-object interaction motions and the corresponding hand masks of the depth images. To obtain the ground truth segmentation masks, we use the aligned color images.

We collect depth images of hands using Intel RealSense SR300. We keep the max depth image size of $640\times 480$ for high-resolution requirements. We record images under a fixed illumination intensity and color temperature for a better segmentation of the color images.

Figure 2 shows the procedure of our auto-annotation for building the ground truth of depth image segmentation.

Background Elimination. After aligning the color and depth images, we crop the hand area in a fixed depth range of 16cm and discard the background. Here we assume that the hand and object are the nearest parts to the camera.

Segmentation. Color space thresholding is able to segment skin from different colored objects. Here we convert color images into HSV space and analyze the linear thresholding restrictions. To reduce noises, we adopt morphological filtering on binary masks of hand and objects respectively. Apart from that, as the alignment cannot be perfect, we adopt a contour filtering step:

[TABLE]

Where $M_{i}$ and $M_{o}$ are the input and output masks. $s$ is a segment in the input mask, and $\mathcal{T}$ computes all the segments. The area and perimeter of a segment are represented by $N(s)$ and $\partial N(s)$. In our implementation, $N(s)$ is measured by the number of pixels in $s$ while $\partial N(s)$ is the number of boundary pixels. $\theta$ and $\phi$ are the two thresholds.

Finally, we check the segmentation results for each frame and discard frames with visible errors, which is greatly faster than pixel-wise manual annotation. The experiment shows over 88% of our auto-annotation results are considered to be acceptable.

We collect a total of 51,912 pairs of depth maps and corresponding ground truth masks with 28 different interacting objects. The whole dataset is split into 46,720 pairs for training and 5,192 for validation. We collect the data in a third person perspective and Table 1 shows the statistics of our dataset as well as some other hand segmentation datasets. Figure 3 shows our high resolution of hand area compared to the recent work [2].

4 DenseAttentionSeg

As shown in Figure 4, we propose DenseAttentionSeg to achieve real-time hand-object segmentation from depth input, which is based on a popular structure of encoder-decoder used in many segmentation tasks. To be specific, we use the same encoder structure in DeepLabv3+ [20]. We also find it works well in our experiments. Meanwhile, we add a decoder that is powered by a densely connected attention structure to fuse all scales of features learned by the encoder and reconstruct the segmentation masks step by step.

4.1 Encoder

The encoder is based on DeepLabv3+ with stride-16 Resnet-50[21] as the network backbone, together with Atrous Spatial Pyramid Pooling(ASPP). We do not use more powerful Resnet-101 and Xception as expressed in [20] for the consideration of real-time performance.

As the network goes deeper, the stride of feature maps gets bigger while the size gets smaller. We choose some intermediate feature maps as different levels, which starts from level 1 to level $n$. Specifically, we select 5 intermediate feature maps in DeepLabv3, together with the final output of ASPP, thus $n=6$.

4.2 Dense Attention Mechanism

We propose an attention mechanism to guide each level to adjust features according to the information of different scales. Figure 5 shows this mechanism in detail. We call it dense because each attention of every level is fused from all the other levels.

Assuming that we are manipulating level $i$, the lower levels of $i-1,i-2...,1$ are gathered through element-wise multiplication step by step, and used to build the fine attention, which conveys the shape and edge feature information to level $i$. Since lower levels have different number of feature channels and larger size than level $i$, all the lower levels are squeezed by a SqueezeNet and down-sampled by a bilinear down-sampling layer before the element-wise multiplication. Similarly, bilinear up-sampling is applied to level $i+1,i+2...,n$ and then the coarse attention is constructed, which is considered to convey semantic feature information.

The SqueezeNet can squeeze the channels of the input feature map to $k$, which not only reduces computation costs but also unifies channels for different levels. A SqueezeNet contains a 1x1 and a 3x3 convolution layer, both with output channels of $k$.

4.3 Decoder

The decoder is applied to reform final segmentation label step by step. We adopt as many as 6 deconvolution[22] layers because we aim to segment hands in a close, fine scale and our operating image size is as big as $640\times 480$. This structure is proved to have higher accuracy than those with fewer decoder layers on our dataset, namely Deeplabv3+.

4.4 Loss Function

We adopt softmax cross entropy loss as our basic training loss, as most of the other segmentation works do.

[TABLE]

In addition, we adopt a contour loss to refine the edges of hands and objects. First, extract edges of both logits of the network output and ground truth labels using Sobel operator, which can be accomplished via convolution layers. A Sobel operation is defined as follows:

[TABLE]

Where $*$ indicates convolution operation, $M$ is the input map with $n$ channels representing different classes, and $M_{l}$ is the $l$th channel of $M$ indicating class $l$. We ignore background class whose label is 0 because it generates inconspicuous contours. The $K_{h}$ and $K_{v}$ are the horizontal and vertical kernel of the Sobel operator respectively. Then, blur the two contour maps using a Gaussian kernel, which can expand the influence of hand and object’s edges smoothly as follows:

[TABLE]

Where $G$ is a 5x5 Gaussian kernel with $\sigma=2.121$. The final contour loss is defined as an L2 loss below:

[TABLE]

Where $M_{labels}$ is the ground truth labels and $M_{logits}$ is the output of our network. The equation sums up all pixels in all label classes and $N$ is the normalization parameter. The combined loss is computed as follows:

[TABLE]

Where $\alpha$ and $\beta$ are the loss weights. We use it as a fine-tuning loss after the training of $\mathcal{L}_{softmax}$ finishes because the contour is not obvious at first.

4.5 Training Details

During training, we apply data augmentation by random flipping and rotation. We also add a Gaussian noise to the input depth maps. The network is trained with a batch size of 4. We use Adam optimizer[23] with an initial learning rate of 0.001, which decays by 0.8 every 80k steps to limit the highest value. The training is almost done after 100 epoch but we stop at 140 epoch to achieve every tiny improvement. Another 10 epoch is added to the fine-tuning procedure of the contour loss.

We set a high softmax loss weight of 5 for hand and object while keep that for the background as 1. The batch-norm loss weight is 0.001, and weights $\alpha$, $\beta$ in contour loss are 1.0 and 0.005. The output channel $k$ in SqueezeNet is set to 64.

5 Experiments

In this section, we first introduce the evaluation metrics. Then we quantitatively and qualitatively compare our method with the state-of-the-art models on the task of general semantic segmentation. Following, we evaluate the contour loss and the dense attention mechanism. After demonstrating the generalization and application of our technique, we introduce the current limitations of our model.

5.1 Evaluation Metrics

Our method aims to segment all pixels in a depth image into three classes - hand, object and background. To evaluate the method, we use the four metrics in [3], which are pixel accuracy, mean accuracy, mean IU and frequency weighted IU. Considering that our objective is to segment hands from objects, we also compute the mean IOU (Intersection over Union) for the hand and object classes.

5.2 Segmentation Results

We train different models on the training set of our InterSegHands and evaluate them on the validation set. The result is shown in Table 2. Our DenseAttentionSeg with contour loss finetuning gains 93.94% mean IOU, which is 1.18% higher than the baseline method DeepLabv3+(Resnet-50, OS-16)[20], and also outperforms other state-of-the-art methods, while still keeps high inference speed.

Besides quantitative evaluations, we also perform some qualitative comparisons in Figure 6. The light-weight FCN-8s fails to distinguish hand and object regions in some examples and cannot identify clear boundaries in some local regions. Deeplabv3+ is somewhat better but still generates noticeable errors. Our DenseAttentionSeg improves the result a lot. Even for small and heavily occluded objects, our model is still robust and generates promising results.

5.3 Evaluation on Dense Attention Mechanism

Our dense attention mechanism aims to direct the skip-connections between encoder and decoder to merge high-level semantic and low-level detailed edge information. In our experiment, the dense mechanism provides a positive effect on our model. Specifically, when we drop the dense attention mechanism, our model gains a mean IOU of 92.81%, which is still a little higher than our baseline model Deeplabv3+ due to our better feature reforming decoder. After adding the dense attention mechanism, the mean IOU rises by 0.84%. The results are shown in Table 3. We also show the attention maps in Figure 7.

5.4 Evaluation on Contour loss

The quantitative and qualitative comparison of our method with and without the contour loss are also shown in Table 2 and Figure 6. We see that the contour loss further improves the result quality on all the metrics. Visually, we can see that the improvements are majorly lying on local boundaries, which is aligned with the theory. And those boundaries are important for accurate hand reconstruction and hand-based remote manipulation. We visualize the effect of our contour loss in Figure 8, where we perform boundary detection on the ground truth segmentation mask, the results without and with the contour loss. They are extracted as Eq. 5 does.

5.5 Generalization and Application

We show some actual testing samples in Figure 9. Our model structure and training strategy make it robust when applied to situations out of our dataset.

6 Conclusion

We have proposed a 52K fine-scale hand-object segmentation dataset based on depth images - InterSegHands Dataset, and a deep model - DenseAttentionSeg to perform hand-object segmentation of interacting motions. Our DenseAttentionSeg contains a dense attention mechanism to fuse all levels features as attention to guide the skip-connection between encoder and decoder, and a contour loss which helps to train the model to get more precise results on hand and object boundaries. According to our evaluations, our DenseAttentionSeg achieves the state-of-the-art performance among several advanced models on our dataset.