ChaLearn Looking at People: IsoGD and ConGD Large-scale RGB-D Gesture Recognition

TL;DR

This paper presents large-scale RGB-D gesture recognition datasets, analyzes current methods, introduces a new segmentation metric, and proposes a Bi-LSTM baseline that outperforms existing techniques.

Contribution

Creation of benchmark datasets for isolated and continuous gesture recognition, analysis of state-of-the-art methods, and introduction of a Bi-LSTM baseline with improved segmentation performance.

Findings

Bi-LSTM baseline outperforms existing methods by 8.1% in CSR

New datasets facilitate large-scale gesture recognition research

Analysis highlights challenges and progress in RGB-D gesture recognition

Abstract

The ChaLearn large-scale gesture recognition challenge has been run twice in two workshops in conjunction with the International Conference on Pattern Recognition (ICPR) 2016 and International Conference on Computer Vision (ICCV) 2017, attracting more than $200$ teams round the world. This challenge has two tracks, focusing on isolated and continuous gesture recognition, respectively. This paper describes the creation of both benchmark datasets and analyzes the advances in large-scale gesture recognition based on these two datasets. We discuss the challenges of collecting large-scale ground-truth annotations of gesture recognition, and provide a detailed analysis of the current state-of-the-art methods for large-scale isolated and continuous gesture recognition based on RGB-D video sequences. In addition to recognition rate and mean jaccard index (MJI) as evaluation metrics used in our previous challenges, we also introduce the corrected segmentation rate (CSR) metric to evaluate the performance of temporal segmentation for continuous gesture recognition. Furthermore, we propose a bidirectional long short-term memory (Bi-LSTM) baseline method, determining the video division points based on the skeleton points extracted by convolutional pose machine (CPM). Experiments demonstrate that the proposed Bi-LSTM outperforms the state-of-the-art methods with an absolute improvement of $8.1\%$ (from $0.8917$ to $0.9639$) of CSR.

Tables

Table 1. TABLE I: Comparison of the RGB-D gesture datasets. The numbers in parentheses correspond to (average per class-minimum per class-maximum per class).

Dataset	Total	Gesture	Avg. samp.	Train samp.
	gestures	labels	per cls.	(per cls.)
CGD [22], 2011	540,000	$>$200	10	8$\sim$12 (1-1-1)
Multi-modal Gesture	13,858	20	692	7,754
Dataset [23], 2013
ChAirGest [24], 2013	1,200	10	120	-
Sheffield Gesture	1,080	10	108	-
Dataset [25], 2013				-
EgoGesture	24,161	83	291	14416
Dataset [26], 2017
IsoGD (Ours)	47,933	249	192	35,878
				(144-64-851)
ConGD (Ours)	47,933	249	192	30,442
				(122-54-722)

Table 2. TABLE II: Summary of both IsoGD and ConGD datasets. Both datasets have been divided into three sub-datasets including different subjects (user independent task).

Sets	the IsoGD dataset				the ConGD dataset
	#labels	#gestures	#RGB-D videos	#subjects	#labels	#gestures	#RGB-D videos	#subjects
Train	249	35878	35878	17	249	30442	14134	17
Validation	249	5784	5784	2	249	8889	4179	2
Test	249	6271	6271	2	249	8602	4042	2
All	249	47933	47933	21	249	47933	22535	21

Table 3. TABLE III: Summary of participation for both gesture challenges (round 1 1 1 and 2 2 2 ).

Challenge	#Round	#Teams1	#Teams2
Isolated Gesture	1	67	20
Challenge	2	44	17
Continuous Gesture	1	66	6
Challenge	2	39	11

Table 4. TABLE IV: Summary of the results in the isolated and continuous gesture recognition challenges.

	rank by test	Team	evaluation
			valid	test
isolated gesture recognition			recognition rate $r$
round 2	1	ASU[33]	64.40%	67.71%
	2	SYSU_ISEE	59.70%	67.02%
	3	Lostoy	62.02%	65.97%
	4	AMRL[34]	60.81%	65.59%
	5	XDETVP[35]	58.00%	60.47%
	-	baseline [36]	49.17%	67.26%
round 1	1	FLiXT[37]	49.20%	56.90%
	2	ARML[38]	39.23%	55.57%
	3	XDETVP- TRIMPS[39]	45.02%	50.93%
	-	baseline[31]	18.65%	24.19%
continuous gesture recognition			MJI
round 2	1	ICT_NHCI[15]	0.5163	0.6103
	2	AMRL[40]	0.5957	0.5950
	3	PaFiFA[41]	0.3646	0.3744
	4	Deepgesture[42]	0.3190	0.3164
round 1	1	ICT_NHCI[16]	0.2655	0.2869
	2	TARDIS[18]	0.2809	0.2692
	3	AMRL[17]	0.2403	0.2655
	-	baseline[31]	0.0918	0.1464

Table 5. TABLE V: Review of state-of-the-art methods on IsoGD and ConGD datasets. It shows that our proposed algorithm achieves the best performance for both datasets and different evaluation metrics (recognition rate, MJI and CSR: the higher the better).

Method	pre-processing	model	fusion strategy	modality of data	evaluation
					valid		test
Isolated gesture recognition evaluated on the IsoGD dataset					Recognition rate
Wan et al.[64, 31]’16	/	MFSK+BoVW	SVM	RGB-D	18.65%		24.19%
Li et al.[37] ’16	32-frame sampling	C3D	SVM	RGB-D	49.20%		56.90%
Wang et al.[38] ’16	bidirectional rank pooling	VGG-16	score fusion	depth (DDI+ DDNI+DDMNI)	39.23%		55.57%
Zhu et al.[39] ’16	32-frame sampling	pyramidal C3D	score fusion	RGB-D	45.02%		50.93%
Zhu et al.[46] ’17	32-frame sampling	C3D, convLSTM	score fusion	RGB-D	51.02%		/
Wang et al.[52] ’17	calibration	AlexNet	score fusion	RGB-D(SFAM)	36.27%		/
Li et al.[47] ’17	32-frame sampling	C3D	SVM	RGB-D saliency	52.04%		59.43%
Li et al.[48] ’17	32-frame sampling	C3D	SVM	RGB-D flow	54.50%		60.93%
Miao et al.[33] ’17	Retinex, median filter, 32-frame sampling	ResC3D	SVM	RGB-D flow	64.40%		67.71%
Wang et al.[40] ’17	bidirectional rank pooling	convLSTM, Resnet-50, C3D	score fusion	RGB-D saliency	60.81%		65.59%
Zhang et al.[35] ’17	32-frame sampling	convLSTM, C3D	score fusion	RGB-D flow	58.00%		60.47%
Zhang et al.[45] ’17	/	AlexNet	score fusion	depth (eDMM+SPM)	36.63%		43.91%
Duan et al.[36] ’17	32-frame sampling	2S CNN, C3D	score fusion	RGB-D saliency	49.17%		67.26%
Hu et al.[65] ’18	32-frame sampling	DNN	adaptive hidden layer	RGB-D	54.14%		/
Wang et al.[58] ’18	bidirectional rank pooling	c-ConvNet	score fusion	RGB-D (VDI+DDI)	44.80%		/
Lin et al.[60] ’18	32-frame sampling	Skeleton LSTM, C3D	adaptive weight fusion	RGB-D Skeleton	64.34%		68.42%
Wang et al.[66] ’18	bidirectional rank pooling	convLSTM, Resnet-50, C3D	score fusion	RGB-D	43.72%		59.21%
Narayana et al. [62] ’18	10-frame sliding window	ResNet-50	Stacked attention models	RGB-D Flow	80.96%		82.07%
Zhu et al. [67] ’19	hand segmentation	shape representaiton	DTW	RGB-D	-		60.12%
Continuous gesture recognition evaluated on the ConGD dataset					MJI	CSR(IoU=0.7)	MJI	CSR(IoU=0.7)
Wan et al.[64, 31]’16	temporal segmentation	MFSK+BoVW	SVM	RGB-D	0.0918	/	0.1464	/
Wang et al.[17] ’16	temporal segmentation	CNN	/	depth (IDMM)	0.2403	0.6636	0.2655	0.7520
Chai et al.[16] ’16	temporal segmentation	2S-RNN, Faster R-CNN, LSTM	/	RGB-D	0.2655	/	0.2869	0.3213
Camgoz et al.[18] ’16	/	C3D	/	RGB	0.3430	/	0.3148	0.6603
Pigou et al.[42] ’17	/	Resnet, LSTM	score fusion	gray-scale	0.3190	0.6159	0.3164	0.6241
Camgoz et al.[41] ’17	temporal segmentation	C3D	/	RGB	0.3806 1	0.8213	0.3744	0.8254
Wang et al.[34] ’17	bidirectional rank pooling, temporal segmentation	convLSTM, C3D	score fusion	RGB-D (DDI+DRI)	0.5957	0.6636	0.5950	0.7520
Liu et al.[15] ’17	calibration, 32-frame sampling, temporal segmentation	C3D, Faster R-CNN	SVM	RGB-D	0.5163	0.9034	0.6103	0.8917
Zhu et al.[68] ’18	temporal segmentation	3D CNN, ConvLSTM	score fusion	RGB-D Flow	0.5368	0.8553	0.7163	0.8776
Bi-LSTM (Ours)	temporal segmentation	C3D,Bi-LSTM [60]	adaptive score fusion [60]	skeleton	0.6830	0.9668	0.7179	0.9639

Table 6. TABLE VI: Temporal segmentation (CSR) methods comparison on validation and test sets on ConGD dataset.

	Validation Set					Testing Set
	0.5	0.6	0.7	0.8	0.9	0.5	0.6	0.7	0.8	0.9
Wang et al.[17]	0.857	0.7954	0.752	0.6997	0.5908	0.7711	0.6963	0.6636	0.6265	0.5497
Chai et al.[16]	-	-	-	-	-	0.709	0.5278	0.3213	0.1458	0.0499
Camgoz et al.[18]	-	-	-	-	-	0.7715	0.7008	0.6603	0.6054	0.5216
Liu et al.[15]	0.9313	0.9122	0.9034	0.8895	0.8132	0.9237	0.9032	0.8917	0.873	0.7750
Wang et al.[40]	0.857	0.7954	0.7520	0.6997	0.598	0.7711	0.6963	0.6636	0.6265	0.5497
pigou et al.[42]	0.7247	0.6625	0.6159	0.5634	0.4772	0.7313	0.6642	0.6241	0.5722	0.4951
Camgoz et al.[41]	0.8856	0.8421	0.8213	0.8024	0.7375	0.8833	0.8441	0.8254	0.8038	0.7187
Bi-LSTM (ours)	0.9784	0.9699	0.9668	0.9638	0.9095	0.9765	0.9686	0.9639	0.9522	0.7876

Equations

$$E_{CSR}(p,l,r)=\frac{\sum _{i=0}^n\sum _{j=0}^{m}M(p_i,l_j,r)}{\max (n,m)}$$

$$M(a,b,r)=\{\begin{array}{ll}1,& \text{IoU}(a,b)\ge r\\ 0,& \text{IoU}(a,b)<r\end{array}$$

$$\text{IoU}(a,b)=\frac{a\cap b}{a\cup b}=\frac{\max (0,\min (a_e,b_e)-\max (a_s,b_s))}{\max (a_e,b_e)-\min (a_s,b_s)}$$

$$V_{t}=\Big{\{}(x_{i}-\bar{x},y_{i}-\bar{y}),i=1,\cdots,60\Big{\}}$$

$$\begin{array}{r}{i}_t=\sigma (W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i)\\ f_t=\sigma (W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f)\\ c_t=f_t{c}_{t-1}+i_t\tanh (W_{xc}{x}_t+W_{hc}{h}_{t-1}+b_c)\\ o_t=\sigma (W_{xo}{x}_t+W_{ho}{h}_{t-1}+b_o)\\ h_t=o_t\tanh (c_t)\end{array}$$

$$y_t$$

$$J(\theta )=-\frac{1}{m}\left[\sum _{c=0}^{m-1}\sum _{i=0}^1{\omega }_i\log \frac{e^{{\theta }_i^T{x}_c}}{{\sum }_{j=0}^k{e}^{{\theta }_j^T{x}_c}}\right],$$

$${▽}_{{\theta }_v}J(\theta )=-\frac{1}{m}\left[\sum _{c=0}^{m-1}\sum _{i=0}^k{\omega }_i{x}_c\left({\mathbb{I}}_{[v=j]}-\frac{e^{{\theta }_i^T{x}_c}}{{\sum }_{j=0}^k{e}^{{\theta }_j^T{x}_c}}\right)\right].$$

Full text

I Introduction

In the past few years, several datasets have been presented to advance the state-of-the-art in computer vision core problems for the automatic analysis of humans, including human pose[1], action/gesture[2, 3, 4, 5, 6], and face[7, 8, 9, 10, 11]. Interestingly, there has been rather limited work focusing on large-scale RGB-D gesture recognition because of the lack of annotated data for that purpose.

To this end, we present two large-scale datasets with RGB-D video sequences, namely, the ChaLearn isolated gesture dataset (IsoGD) and continuous gesture dataset (ConGD) for the tasks of isolated and continuous gesture recognition, respectively. Both of them consist of more than 47 thousands gestures fallen into $249$ classes performed by $21$ performers. The IsoGD dataset has $47,933$ video clips (one gesture per video) whereas the ConGD dataset has $22,535$ clips owing to some continuous gestures existing in one video. Specifically, we developed software to help complete the annotation efficiently. That is, we first used the dynamic time wrapping (DTW) algorithm [12] to approximately determine the start and end points of the gestures, and then manually adjusted the start and end frames of each gesture accurately. Then, we organized two ChaLearn Large-scale Gesture Recognition Challenge workshops in conjunction with the ICPR 2016 [13], and ICCV 2017 [14]. The datasets allowed for the development and comparison of different algorithms, and the competition and workshop provided a way to track the progress and discuss the advantages and disadvantages learned from the most successful and innovative entries.

We analyze and review the published papers focusing on large-scale gesture recognition based on IsoGD or ConGD datasets, introduce a new CSR metric evaluation and propose a baseline method for temporal segmentation. Specifically, instead of deploying unreliable prior assumptions [15, 16, 17] and handling each video frame separately [18], we design a new temporal segmentation algorithm to convert the continuous gesture recognition problem to the isolated one, by using the bidirectional long short-term memory (Bi-LSTM) to determine the video division points based on the skeleton points extracted by the convolutional pose machine (CPM) [19, 20, 21] in each frame.

In addition, we also discuss and analyze the achieved results and propose potential directions for future research. We expect the challenge to push the progress in the respective fields. The main contributions of this work are summarized as follows:

• •

We discuss the challenges of creating two large-scale gesture benchmark datasets, namely, the IsoGD and ConGD, and highlight developments in both isolated and continuous gesture recognition fields by creating the benchmark and holding the challenges. We analyze the submitted results in both challenges, and review the published algorithms in the last three years.

• •

A new temporal segmentation algorithm named the Bi-LSTM segmentation network is proposed, which is used to determine the start and end frames of each gesture in the continuous gesture video. Compared with existing methods for temporal segmentation, the main advantage of the proposed method is to avoid the need for prior assumptions.

• •

A new evaluation metric named corrected segmentation rate (CSR) is introduced and used to evaluate the performance of temporal segmentation. Compared with the published methods, the proposed Bi-LSTM method improves state-of-the-art results. The superiority of temporal segmentation is about 8.1% (from 0.8917 to 0.9639) by CSR on the testing sets of the ConGD dataset.

The rest of this paper is organized as follows. We describe datasets, evaluation metrics and organized challenges in Section II. In Section III, we review the state-of-the-art methods focusing on both datasets, and analyze the results in detail. We propose a new algorithm for temporal segmentation in Section IV and present experimental results on the two proposed datasets in Section V. Finally, we conclude the paper and discuss some potential research directions in the field in Section VI.

II Dataset Introduction and Challenge Tasks

II-A Motivation

Benchmark datasets can greatly promote the research developments in their respective fields. For example, the ImageNet Large Scale Visual Recognition Challenge [27] (ILSVRC) is held every year from 2010 to 2017, which includes several challenging tasks, including image classification, single-object localization, and object detection. The dataset presented in this challenge contains $1000$ object classes with approximate $1.2$ million training images, $50$ thousand validation images and $100$ thousand testing images, which greatly promotes the development of new techniques, particularly those based on deep learning architectures, for image classification and object localization. Several other datasets have been also designed to evaluate different computer vision tasks, such as human pose recovery, action and gesture recognition, and face analysis [23, 28, 29, 30].

Nevertheless, there are very few annotated datasets with a large number of samples and gesture categories for the task of RGB-D gesture recognition. Table I lists the publicly available RGB-D gesture datasets released from 2011 to 2017. Most datasets include less than $20$ gesture classes (e.g., [24, 25]). Although the CGD dataset [22] has about $54$ thousand gestures, it is designed for one-shot learning task (only one training sample per class). The multi-modal gesture dataset [23, 28] contains about $13$ thousand gestures with $387$ training samples per class, but it only has $20$ classes.

In order to provide to the community with a large dataset for RGB-D gesture recognition, here we take benefit of the previous CGD dataset [22] by integrating all gesture categories and samples to design two new large RGB-D datasets for gesture spotting and classification. In Table I, the new IsoGD and ConGD datasets show a significant increase in size in terms of both the number of categories and the number of samples in comparison to state-of-the-art alternatives.

II-B Data Collection Procedure

As aforementioned, our datasets were derived from the CGD dataset [22] which was designed for the ”one-shot learning” task. Therefore, we first introduce the CGD dataset. As shown in the left of Fig. 1, the CGD dataset contained 540 batches (or subfolders). Each batch was designed for a specific ”one-shot learning” task, where it had a small gesture vocabulary, 47 RGB-D video clips including 100 gestures in total, and only one training sample per each class. It was independent among different batches. All batches of CGD had 289 gestures from 30 lexicons and a large number of gestures in total (54000 gestures in about 23000 RGB-D video clips), which makes it good material to carve out different tasks. This is what we did by creating two large RGB-D gesture datasets: the IsoGD111http://www.cbsr.ia.ac.cn/users/jwan/database/isogd.html and ConGD222http://www.cbsr.ia.ac.cn/users/jwan/database/congd.html datasets. The process of data annotation from the CGD dataset is shown in Fig. 1. It mainly includes two steps: semi-manual temporal segmentation and gesture labelling.

In the stage of temporal segmentation, we used a semi-manual software to annotate the begin and end frames of each gesture in video sequences. Later, we checked all 540 batches with its corresponding lexicons, and merged the similar gesture (see one sample in Fig. 2) in different lexicons manually. For detailed information about gesture annotation, the reader is referred to [31].

Finally, we obtained $249$ unique gesture labels, $47,933$ gestures in $22,535$ RGB-D videos, which are derived from $480$ batches of the CGD dataset. Some samples of dynamic gestures from both datasets are shown in Fig. 3.

II-C Dataset Statistics

The statistical information of both gesture datasets is shown in Table II. For the ConGD dataset, it includes $47,933$ RGB-D gestures in $22,535$ RGB-D gesture videos. Each RGB-D video can represent one or more gestures, and there are $249$ gestures labels performed by $21$ different. For the IsoGD dataset, we split all the videos of the CGD dataset into isolated gestures, obtaining $47,933$ gestures. Each RGB-D video represents one gesture instance, having $249$ gestures labels performed by $21$ different individuals.

II-D Evaluation Metrics

For both datasets, we provide training, validation, and test sets. In order to make it more challenging, all three sets include data from different subjects, which means the gestures of one subject in validation and test sets will not appear in the training set. According to [31], we introduced the recognition rate $r$ and mean jaccard index (MJI) $\bar{J}_{acc}$ as the evaluation criteria for the IsoGD and ConGD datasets, respectively.

The MJI $\bar{J}_{acc}$ is a comprehensive evaluation on the final performance of continuous recognition, but it does not give a specific assessment on either the classification or the segmentation. This drawback makes it difficult to tell whether a high-performance method is attributed to its classification or segmentation strategy.

Therefore, to evaluate the performance of temporal segmentation, we present the corrected segmentation rate (CSR) $E_{CSR}$, based on intersection-over-union (IoU). $E_{CSR}$ is defined as:

[TABLE]

where $p$ is the target model’s predicted segmentations for each video, which is constituted by the position of the starting and ending frames. $l$ is the ground truth, which has the same form as $p$. $n$ and $m$ are the number of segmentations in the model’s prediction and the ground truth, respectively. $M$ is the function to evaluate whether the two sections are matched or not with a predefined threshold $r$:

[TABLE]

where $a$ is the segmentation result that needs evaluation and $b$ is the ground truth. The IoU function is defined below, which is similar to its definition for object detection [32].

[TABLE]

where $a_{s}$, $a_{e}$ represent the starting frame, and the ending frame of the segmentation $a$. $b_{s}$ and $b_{e}$ are in a manner analogous to $a_{s}$ and $a_{e}$. If IoU(a, b) is greater than the threshold $r$, we consider they are matched successfully.

II-E Challenge Tasks

Both large-scale isolated and continuous gesture challenges belong to the series of ChaLearn LAP events333http://chalearnlap.cvc.uab.es/, which were launched in two rounds in conjunction with the ICPR (Cancun, Mexican, December, 2016) and ICCV (Venice, Italy, October, 2017). This competition consisted of a development phase (June 30, 2016 to August 7, 2016 for the first round, April 20, 2017 to June 22, 2017 for the second round) and a final evaluation phase (August 7, 2016 to August 17, 2016 for the first round, June 23, 2017 to July 2, 2017 for the second round). Table III shows the summary of the participation for both gesture challenges. The total number of registered participants of both challenges is more than 200, and 54 teams have submitted their predicted results.

For each round, training, validation and test data sets were provided. Training data were released with labels, validation data were used to provide feedback to participants in the leaderboard and test data were used to determine the winners. Note that each track had its own evaluation metrics. The four tracks were run in the CodaLab platform444https://competitions.codalab.org/. The top three ranked participants for each track were eligible for prizes. The performances of winners are shown in Table IV.

Development phase.

Participants had access to labeled development (training) and unlabeled validation data. During this phase, participants received immediate feedback on their performance on validation data through the leaderboard in CodaLab.

Final phase.

The unlabeled final (test) data were provided. The winners of the contest were determined by evaluating performances on these two datasets. The participants also had to send code and fact sheets describing their methods to challenge organizers. All the code of participants was verified and replicated prior to announcing the winners. All the test labels of both datasets are released nowadays, which can be found on the website of IsoGD and ConGD.

The challenge tasks proposed were both ”user independent” and consist of:

• •

Isolated gesture recognition for the IsoGD dataset.

• •

Gesture spotting and recognition from continuous videos for the ConGD dataset.

As shown in Table II, the datasets were split into three subsets: training, validation, and test. The training set included all gestures from 17 subjects, the validation set included all gestures from 2 subjects, and the rest gestures from 2 subjects were used in the test set. We guaranteed that the validation and test sets included gesture samples from the $249$ labels.

III Review of State-of-the-art Methods

In recent years, the commercialization of affordable RGB-D sensors, such as Kinect, made it available depth maps, in addition to classical RGB, which are robust against illumination variations and contain abundant 3D structure information. Based on this technology, we created the IsoGD and ConGD datasets [31], which has been already used by several researchers to evaluate the performance of gesture recognition models. In this section, we review and compare these methods.

As illustrated in Fig. 4, current methods can fall into two categories according to whether they address isolated or continuous gesture recognition. Since gestures in the isolated gesture dataset are separated in advance, the main concern is how to issue a label to a certain gesture. Due to the promising achievement in the fields of object recognition, deep convolutional networks are also considered as the first choice for gesture recognition. The 2D CNN is very common in practice[43, 44]. In order to make a trade-off between the 2D network and spatiotemporal features, techniques like rank pooling are employed to involve the temporal information in an image like dynamic depth image[17, 38, 45, 40, 34, 14]. Some methods [37, 18, 39, 15, 33, 34, 35, 46, 47, 48] use the 3D CNN model [49] to learn spatiotemporal features directly and concurrently. Meanwhile, the RNN [50] or its variation LSTM [51] are also applied to analyze sequential information from input videos [16, 40, 42, 35, 46]. However, the task for continuous gesture recognition can be more arduous. There may be 4-5 gestures in a video, so it is necessary to recognize which gesture a series of motion belong to. It can be achieved either in a frame-by-frame fashion [18, 41] or by a temporal segmentation beforehand [16, 17, 15, 40].

III-A Pre-processing

Although deep learning methods work well on the gesture recognition challenge, the quality of data still plays an important role. Therefore pre-processing is commonly employed by many methods [37, 52, 33, 15, 53]. There are two main categories of pre-processing. The first category of pre-processing is video enhancement. Since the videos are obtained under different conditions, the RGB videos are prone to be influenced by illumination changes. Its counterpart, the depth videos, are insensitive to illumination variations but presented some noise. Miao et al.[33] implement the Retinex theory [54] to normalize the illumination of RGB videos, and use the median filter to denoise depth maps. Asadi et al.[53] utilize a hybrid median filter and inpainting technique to enhance depth videos. The second category of pre-processing is based on frame unification and video calibration. The reason for frame unification is to fix the dimension of all inputs for the fully-connected layers in CNNs. After a statistical analysis of frame number distribution of training data on the IsoGD dataset, Li et al.[37] fix the frame number of each clip as 32 to minimize the loss of motion path in the temporal dimension. The same criterion has been used by most subsequent methods [15, 33, 46, 47, 48, 35]. Meanwhile, although the RGB and depth videos are captured concurrently by a Kinect sensor, ensuring temporal synchronization, they are not often registered spatially. Such a spatial misalignment may affect the multi-modality fusion. Therefore, Wang et al.[52] propose a self-calibration strategy based on a pinhole model to register those data. A similar way is also conducted by Asadi et al.[53], which exploits the intrinsic and extrinsic parameters of cameras to warp the RGB image to fit the depth one.

Temporal segmentation can also be deemed as a kind of pre-processing method, which is only applied for continuous gesture recognition. Since there is more than one gesture in the video for continuous tasks, researchers should pay more attention to separating the gestures from each other. One possibility is dividing the videos into several clips containing only one gesture each, which can then analyzed as the isolated gesture recognition task. Chai et al.[16] first take such a segmentation strategy for continuous gesture recognition. It assumes all gestures begin and end with performers’ hands down. Then the video can be characterized as successive gesture parts and transition parts. A similar idea is used in Wang et al.[17], Liu et al.[15] and Wang et al.[40]. Camgoz et al.[41] conduct such a temporal segmentation in a different way. They treat the segmentation process as a feature to learn, and use the likelihood to split the videos into multiple isolated segments, which is done by localizing the silence regions, where there is no motion.

III-B Network models

Network models are the key part of gesture recognition. Common models include 2D CNNs, 3D CNNs, RNN/LSTM, and detection models such as Faster R-CNN.

2D CNNs.

2D CNN models like AlexNet [43], VGG [44] and ResNet [55] have shown great performance dealing with recognition tasks. There are several methods [56, 38, 52] that first implement the 2D CNN to extract gesture features. However, in its standard way, 2D CNN convolution and pooling only manipulate the spatial dimension, not considering temporal data dynamics. In order to extend 2D CNN to consider temporal information, Wang et al.[38] use rank pooling [57] to generate dynamic depth images (DDIs), and compute dynamic depth normal images (DDNIs) and Dynamic Depth Motion Normal Images (DDMNIs) to wrap both the motion information and the static posture in an image. A similar idea is utilized by Wang et al.[40]. The counterpart, visual dynamic images (VDIs) for RGB videos, is generated in [58]. Moreover, Wang et al.[34] extend the DDIs for both body level and hand level representation, which are called body level Dynamic Depth Images (BDDIs) and Hand Level Dynamic Depth Images (HDDIs), respectively. Zhang et al.[45] use an enhanced Depth Motion Map (eDMM) to describe depth videos and a Static Pose Map (SPM) for postures. Then two CNNs are used to extract features from these representations. Wang et al.[52] use the scene flow vector, which is obtained by registered RGB-D data, as a descriptor to generate an action map, which is subsequently fed into AlexNet for classification.

3D CNNs.

3D CNNs like C3D [49] were proposed to extend 2D CNNs to compute spatiotemporal features. Li et al.[37, 47, 48] utilize 3D CNN to extract features from RGB-D, saliency, and optical flow videos. Zhu et al.[39, 46] propose a pyramid 3D CNN model, in which the videos are divided into three 16 frame clips, performing prediction in each of them. Final recognition is obtained by means of score fusion. Such a pyramid 3D CNN model is also employed by [40, 35, 34]. Liu et al.[15] and Wang et al.[40] use a 3D CNN model in a similar way for continuous gesture recognition. Camgoz et al.[18] also use 3D CNNs for feature extraction. However, aiming at continuous gesture recognition, they issue the label of the clip centered at frame $i$ to exact that frame. Based on this they find all gestures in a video. In [41], the 3D CNN model is still used in a frame-wise fashion, with the final classification given by posterior estimation after several iterations.

RNN/LSTM.

The recurrent neural network (RNN) [50], or its variation, long short-term memory (LSTM), [51] is a kind of network where connections between units form a directed cycle. The special structure of RNN-like models allows for sequential input analysis. Chai et al.[16] use two streams of RNN to represent features of RGB-D videos, and use LSTM to model the context. Pigou et al.[42] first use a ResNet to extract features of gray-scale video, and then use a bidirectional LSTM [59] to process both temporal directions. Zhu et al.[46] use convLSTM with 3D CNN input to model the sequential relations between small video clips. The 3D CNN + LSTM scheme is also employed in [35, 40, 60].

Faster R-CNN.

The faster R-CNN [61] was initially proposed for detection tasks. Since the gestures are often started and ended with hands down, the detected location of hands is used to indicate the interval of gestures in a continuous video. Chai et al.[16] use faster R-CNN to help to segment gestures. This strategy is widely applied for continuous gesture recognition [15, 40]. In addition, some methods complemented with hand region detection to further boost recognition performance[34, 14].

Attention Models.

Attention-aware methods [62, 63] have also been applied for gesture recognition. Pradyumna et al. [62] proposed a focus of attention network (FOANet) which introduced a separate channel for every focus region (global, right/left hand) and modality (RGB, depth and flow). Zhang et al.. [63] proposed an attention mechanism embedding into the convolutional LSTM (ConvLSTM) network, including attention analysis in ConvLSTM gates for spatial global average pooling and fully-connected operations.

III-C Multi-Modality Fusion Scheme

Since IsoGD and ConGD datasets include two visual modalities, fusion of RGB and Depth data uses to be considered for an enhanced recognition performance. Score fusion is a popular strategy [38, 39, 34, 40, 46, 58]. This kind of scheme consolidates the scores generated by networks which are fed with different modalities. Among these methods, the averaging [39, 34, 40, 46] and multiply [38, 58] score fusions are two of the most frequently applied. Li et al. [37, 47, 48], Zhu et al. [39], and Miao et al. [33] adopted feature level fusion. The former methods [37, 39] directly blend the features of RGB and depth modalities in a parallel or serial way, which simply average or concatenate. Considering the relationship between features from different modalities that share the same label, Miao et al. [33] adopt a statistical analysis based fusion method - canonical correlation analysis, and Li et al. [48, 47] adopt an extension version of discriminative correlation analysis, which tries to maximize the inner-class pair-wise correlations across modalities and intra-class differences within one feature set. Hu et al.[65] pay more attention to the fusion scheme and design a new layer comprised of a group of networks called adaptive hidden layer, which serves as a selector to weight features from different modalities of data. Lin et al.[60] developed an adaptive scheme for setting weights of each voting sub-classifier via a fusion layer, which can be learned directly by the CNNs.

III-D Other Techniques to Boost Performance

Multiple modalities.

Based on the available RGB and depth data modalities in the proposed datasets, additional data modalities have been considered by researchers. Li et al.[47] generate saliency maps to focus on image parts relevant to gesture recognition. Then also use optical flow data [48] to learn features from image motion vectors. Wang et al.[52] and Asadi et al.[53] notice the drawbacks of optical flow, which can only be used for constructing 2D motion information, and use scene flow, which considers the motion of 3D objects by the combination of RGB and depth data. There are also some methods that employ skeleton data using Regional Multi-person Pose Estimation (RMPE)[69, 14].

Data augmentation.

Data augmentation is another common way to boost performance. Miao et al.[33] focuses on data augmentation to increase overall dataset size while Zhang et al.[45] mainly augment data to balance the number of samples among different categories. Their augmentation tricks include translation, rotation, Gaussian smoothing and contrast adjustment.

Pre-trained models.

Finetuning with models pre-trained on large datasets uses to be considered to reduce overfitting effect. Most C3D-implemented methods are pre-trained on the sports-1M dataset [70]. In terms of cross-modality finetuning, Zhu et al.[46] first train the networks with RGB and depth data from scratch, and then finetune the depth ones with the models of the RGB counterpart. The same process is done on the RGB network. The result of cross-modality finetuning showed an improvement of 6% and 8% for RGB and depth inputs, respectively.

III-E State-of-the-art comparison

According to the previous analysis, here we provide with a state-of-the-art comparative on both IsoGD and ConGD datasets. Considered methods were published in the last three years and results are shown in Table V. On the IsoGD dataset, 2D/3D CNN have been widely used, and the recognition rate has been improved by 58% (from 24.19% [31] to 82.17% [62]). Owing to the difficult task for continuous gesture recognition, only a few papers have considered the ConGD dataset. However, the performance has been also improved greatly on the metric of both MJI and CSR since 2017. In Table V, the performance of the proposed Bi-LSTM method is shown, which is further discussed in Section V.

III-F Discussion

In this section, we review the techniques on both isolated and continuous gesture recognition based on RGB-D data. After the release of the large-scale IsoGD and ConGD datasets, new methods have pushed the development of gesture recognition algorithms. However, there are challenges faced by available methods which allow us to outline several future research directions for the development of deep learning-based methods for gesture recognition.

Fusion of RGB-D modalities. Most methods [38, 39, 34, 40, 46] considered RGB and depth modality as a separate channel and fused them at a later stage by concatenation or score voting, without fully exploiting the complementary properties of both visual modalities. Therefore, cooperative training using RGB-D data would be a promising and interesting research direction.

Attention-based mechanism. Some methods [62, 15, 16] used hand detectors to first detect hand regions and then designed different strategy to extract local and global features for gesture recognition. However, these attention-based methods need hard to train specialized detectors to find hand regions properly. It would be more reasonable to consider sequence modeling self-attention [71, 72] and exploit it for dynamic gesture recognition.

Simultaneous gesture segmentation and recognition. Existing continuous gesture recognition works [15, 34, 17, 64] first detect the first and end point of each isolated gesture, and then train/test each segment separately. This procedure is not suitable for many real applications. Therefore, simultaneous gesture segmentation and recognition would be an interesting line to be explored.

IV Temporal Segmentation Benchmark

In this section, we propose a benchmark method, namely Bi-LSTM network, for temporal segmentation. Before introducing the proposed method, we first illustrate drawbacks of the current temporal segmentation methods.

IV-A Drawbacks of temporal segmentation methods

1. Hand-crafted Hand Motion Extraction: Some methods [64, 31, 17, 34] first measure the quantity of movement (QoM) for each frame in a multi-gesture sequence and then threshold the quantity of movement to obtain candidate boundaries. Then, a sliding window is commonly adopted to refine the candidate boundaries to produce the final boundaries of the segmented gesture sequences in a multi-gesture sequence. However, it captures not only the hand motion but also background movements which may be harmful to temporal segmentation.

2. Unstable Hand Detector: Some methods [16, 15] used the Faster R-CNN [73] to build the hand detector. Owing to the high degree of freedom of human hands, it is very hard to tackle some intractable environments, such as hand-self occlusion and drastically hand shape changing. The errors of hand detection would considerably reduce the performance of temporal segmentation.

3. Strong Prior Knowledge Requirement: Most previous methods (e.g., [64, 31, 16, 15, 17]) use prior knowledge (e.g., a performer always raises hands to start a gesture, and puts hands down after performing a gesture). The strong prior knowledge (i.e., hand must lay down after performing another gesture) is not practical for real applications.

In contrast to previous methods, we did not only use human hands but also the arm/body information [16, 15]. Moreover, we designed a Bi-LSTM segmentation network to determine the start-end frames of each gesture automatically without requiring specific prior knowledge.

IV-B the Proposed Bi-LSTM Method

We treat the temporal segmentation as a binary classification problem. The flowchart is shown in Fig. 5. We first use the convolutional pose machine (CPM) algorithm555https://github.com/CMU-Perceptual-Computing-Lab/openpose [19, 20, 21] to estimate the human pose, which consists of $60$ keypoints ($18$ keypoints for human body, $21$ keypoints for left and right hands, respectively). The keypoints are shown in the left part of Fig. 5. Therefore, the human gesture/body from an image is represented by these keypoints. For the $t^{th}$ frame of a video, the gesture is represented by a $120$-dimension ($2\times 60$) vector $V_{t}$ in Eq. 4.

[TABLE]

where the coordinate of the $i^{th}$ keypoint is represented by $(x_{i},y_{i})$, the average coordinate of all detected keypoints is denoted by $(\bar{x},\bar{y})$, and $\bar{x}=\frac{1}{n_{t}^{{}^{\prime}}}\sum_{k=1}^{n_{t}^{{}^{\prime}}}{x_{k}}$, $\bar{y}=\frac{1}{n_{t}^{{}^{\prime}}}\sum_{k=1}^{n_{t}^{{}^{\prime}}}{y_{k}}$, $n_{t}^{{}^{\prime}}$ is the number of detected keypoints of frame $t$.

We use the data $\{(V_{t},g_{t})|t=1,\cdots,m\}$ to train the Bi-LSTM network [59], where $m$ is the total number of frames in the video, and $g_{t}$ is the start and end frames indicator of a gesture, i.e., $g_{t}=1$ indicates the start and last frames of a gesture, and $g_{t}=0$ for other frames. The Bi-LSTM network combines the bidirectional RNNs (BRNNs) [74] and the long short-term memory (LSTM), which captures long-range information in bi-directions of inputs. The LSTM unit $\mathcal{H}$ is implemented by the following composite function:

[TABLE]

where $\sigma_{t}$ is the activation function, $i_{t}$, $f_{t}$, $c_{t}$, $o_{t}$ and $h_{t}$ are the input gate, forget gate, output gate, cell activation vector and the hidden vector at time $t$, respectively. For the Bi-LSTM, the network computes both the forward and backward hidden vectors $\overrightarrow{h}_{t}$ and $\overleftarrow{h}_{t}$ at time $t$, and the output sequence $y_{t}$ as:

[TABLE]

We design four hidden layers in the Bi-LSTM network in Fig. 5. Notably, if a frame is within the segment of a gesture, we annotate it as the positive sample; otherwise, it is treated as negative. To mitigate the class imbalance issue, we assign different weights to the positive and negative samples. The objective function is defined as:

[TABLE]

where $\theta$ is the parameter matrix of the softmax function, and $w_{i}$ is the weight used to mitigate the class imbalance issue. According to our statistics, the ratio of the positive and negative samples is approximately $1:40$. Thus, we set $\frac{w_{0}}{w_{1}}=40$ ($w_{0}$ is the weight penalty of positive samples) to balance the loss terms of positive and negative samples in the training phase.

The gradient of the objective function is computed as

[TABLE]

where $\triangledown_{\theta_{v}}J(\theta)$ is the gradient with respect to the parameter $\theta_{v}$, and $\mathbb{I}_{[v=j]}$ is the indicator function, i.e., $\mathbb{I}_{[v=j]}=1$ if and only if $v=j$.

In this way, we use the learned model by the Bi-LSTM network to predict the probability of each frame whether it is the start or end frames. If the probability value of a frame is large than 0.5, this frame is treated as start or end frames.

V Experiments

In this section, we evaluate and compare our proposed gesture recognition by segmentation strategy on the ConGD dataset. First, the experimental setup is presented, including the running environments and settings. Then, the performances and comparisons on ConGD datasets are given.

Our experiments are conducted on a NVIDIA Titan Xp GPU with PyTorch [75]. For the Bi-LSTM segmentation network, the input is a 120-dimension vector. We use the Adam algorithm [76] to optimize the model with the batch size $120$. The learning rate starts from $0.01$, and the models are trained for up to $50$ epochs.

The performance of the proposed Bi-LSTM method for temporal segmentation is shown in Table VI, which achieves 0.9668 and 0.9639 for CSR@IoU=0.7 on both validation and testing sets of ConGD. After temporal segmentation, we use the model of  [60] to perform final gesture recognition. The results are also shown in Table VI, where MJI$=0.6830$ and $0.7179$ on the validation and test sets, respectively. Based on MJI and CSR, our method ahieves the best performance. Although the metric of MJI depends on both temporal segmentation and final classification, the recognition performance of MJI can still benefit from an accurate temporal segmentation, such as the proposed Bi-LSTM method.

We also provide with a MJI comparison for each category on the ConGD dataset in Fig. 8. Here, our method (overall MJI: 0.6830 for validation set, 0.7179 for test set) is compared with two state-of-the-art methods [15] (overall MJI: 0.5163 for validation set, 0.6103 for test set) and  [68] (overall MJI: 0.5368 for validation set, 0.7163 for test set) for each category. From Fig. 8, compared with [15], one can see that the methods of Zhu [68] and ours achieve high performance in most categories. Our method obtain high performance for all categories, while the other two methods fail recognizing some categories, such as gesture class ID 20, 62 for Liu’s method [15], and gesture class ID 62, 148 for Zhu’s method [68].

Fig. 6 shows the CSR curve in each epoch under different IoU thresholds from $0.5$ to $0.9$. One can see that when the IoU threshold is between $0.5$ and $0.8$, the CSR is very stable after $3$ epochs. When IoU is equal to $0.9$, the training epochs for the CSR increases. This is because the correct condition is more strict ( $>90\%$ overlapped region will be treated as the correct one) and it will cost more time to seek the best CSR. Our proposed Bi-LSTM method can get very stable results under different IoU thresholds. For example, even the IoU is equal to $0.9$, the CSR of our method still is higher than $0.9$. Alternative temporal segmentation methods [17, 15, 42, 41] are relatively inferior (the best is about $0.81$ in [15]) on the validation set. Also, our Bi-LSTM can get the best performance on the test set of the ConGD dataset.

Then, we randomly select 1000 video sequences in the ConGD datasets to check for computational requirements. It required about 0.4 second under GPU environment (NVIDIA TITAN X (Pascal)) and 6 seconds on CPU environment (Intel(R) Core(TM) i7-5820K@3.30GHz) for the proposed Bi-LSTM method (excluding CPM algorithm). It demonstrates the proposed Bi-LSTM method is ultra high-speed processing ($\sim$0.4ms/video-GPU, $\sim$6 ms/video-CPU).

Finally, we selected 3 longest video sequences of the ConGD dataset, and the segmentation results of the proposed Bi-LSTM method are shown in Fig. 7. The green points are the ground truth of the segmentation point, while the blue line is the confidence of positive responses. These three videos have more than 100 frames and contain at least 5 gestures. Compared with the videos with fewer number of gestures, the dense gestures make it hard to find the segment points accurately. However, our Bi-LSTM method can mark the start and end points of each gesture, and the segmentation for all the gestures are with confidence over 0.8.

VI Conclusion

In this paper, we proposed the IsoGD and ConGD datasets for the task of isolated and continuous gesture recognition, respectively. Both datasets are the current largest datasets for dynamic gesture recognition. Based on both datasets, we have run challenges in ICPR 2016 and ICCV 2017 workshops, which attracted more than 200 teams around the world and pushed the state-of-the-art for large-scale gesture recognition. Then, we reviewed last 3-years state-of-the-art methods for gesture recognition based on the provided datasets. In addition, we proposed the Bi-LSTM method for temporal segmentation. We expect the proposed datasets to push the research in gesture recognition.

Although some published papers have achieved promising results on the proposed datasets, there are several venues which can be explored to further improve gesture recognition. First, a structure attention mechanism can be further explored. In our method, each attention part (i.e., arm, gesture) is trained separately. We believe gesture recognition will benefit from joint structure learning (*i.e.*body, hand, arm and face). Second, new end-to-end methods are expected to be designed. We discussed different works that benefited from the fusion and combination of different trained models and modalities. By considering end-to-end learning it is expected to further boost performance. This will also require detailed analyses on computation/accuracy trade-off. Moreover, online gesture recognition is another challenging problem. The research for an efficient gesture detection/spotting/segmentation strategy is an open issue. We expect ConGD dataset to support the evaluation of models in this direction.