KPTransfer: improved performance and faster convergence from keypoint subset-wise domain transfer in human pose estimation

TL;DR

KPTransfer enhances human pose estimation by transferring contextual knowledge between keypoint subsets, leading to improved accuracy and faster training convergence in neural networks.

Contribution

The paper introduces KPTransfer, a novel domain transfer method between keypoint subsets that improves performance and convergence speed in pose estimation models.

Findings

Up to 1.1% PCKh improvement over random initialization.

Overall 0.5% PCKh improvement across joints.

Faster convergence due to mutual co-adaptations of weights.

Abstract

In this paper, we present a novel approach called KPTransfer for improving modeling performance for keypoint detection deep neural networks via domain transfer between different keypoint subsets. This approach is motivated by the notion that rich contextual knowledge can be transferred between different keypoint subsets representing separate domains. In particular, the proposed method takes into account various keypoint subsets/domains by sequentially adding and removing keypoints. Contextual knowledge is transferred between two separate domains via domain transfer. Experiments to demonstrate the efficacy of the proposed KPTransfer approach were performed for the task of human pose estimation on the MPII dataset, with comparisons against random initialization and frozen weight extraction configurations. Experimental results demonstrate the efficacy of performing domain transfer between two different joint subsets resulting in a PCKh improvement of up to 1.1 over random initialization on joints such as wrists and knee in certain joint splits with an overall PCKh improvement of 0.5. Domain transfer from a different set of joints not only results in improved accuracy but also results in faster convergence because of mutual co-adaptations of weights resulting from the contextual knowledge of the pose from a different set of joints.

Figures

Tables

Table 1. Table 1: The subsets considered for experiments

		Subset $S_1$	Subset $S_2$
	(a)	Head, Neck, Shoulders, Pelvis, Thorax, Hip	Knees, Ankles, Wrists, Elbows
	(b)	Head, Neck, Shoulders, Elbows, Hip	Knees, Ankles, Wrists, Pelvis, Thorax
	(c)	Head, Neck, Shoulders, Elbows, Knees	Wrist, Ankles, Hip, Pelvis, Thorax
	(d)	Knees, Ankles, Wrists, Elbows	Head, Neck, Elbows, Knee, Pelvis, Thorax

Table 2. Table 2: PCKh comparison for joint split a

Configuration	Elbow	Wrist	Knee	Ankle	Average
Transfer learning	87.9	84.2	82.9	80.6	83.9
Frozen weights	74.7	56	72.7	66.8	67.5
Random initialization	87.9	83.9	83.4	80.5	83.9

Table 3. Table 3: PCKh comparison for joint split b

Configuration	Wrist	Knee	Ankle	Average
Transfer learning	84.1	84.5	81.5	83.4
Frozen weights	69.7	73.0	62.6	68.43
Random initialization	83.6	85	81.6	83.4

Table 4. Table 4: PCKh comparison for joint split c

Configuration	Wrist	Hip	Ankle	Average
Transfer learning	84	87.1	80.1	83.7
Frozen weights	71.2	83.7	69.9	74.9
Random initialization	82.9	87.0	79.7	83.2

Table 5. Table 5: PCKh comparison for joint split d

Configuration	Head	Elbow	Knee	Average
Transfer learning	96.8	88.3	84.4	89.8
Frozen weights	91.1	87.9	82.9	87.3
Random initialization	97.1	87.9	83.3	89.4

Full text

1 Introduction

In any keypoint estimation problem, the location of a particular keypoint holds contextual information about the location of another. In pose estimation, for example, the position of the elbows and wrists are naturally constrained, being part of the same limb. Deep keypoint estimation algorithms take advantage of this fact by learning different keypoint locations simultaneously. For instance, deep pose estimation algorithms predict human joint locations together [3, 14, 17, 22] or use a two-pipeline framework for body part detection and association [18, 20]. However, domain transfer between keypoints remains an unexplored area. We hypothesize that domain transfer can be used for utilizing the contextual relationships between keypoint locations for improving convergence and generalization. Since a large number of keypoint detection datasets [15, 1, 13] are available which differ in keypoint location annotations, one obvious question arises: Can domain transfer between separate keypoint sets help in improving generalization performance and convergence?

In this paper, we introduce a novel approach termed KPTransfer for performing domain transfer between keypoint subsets representing separate domains. We apply our approach on the problem of 2D human pose estimation. However, our approach is completely task and model agnostic and can be used to evaluate domain transfer using any deep pose estimation model in any other keypoint detection problem such as facial landmark detection and 3D pose estimation. We use the stacked hourglass pose estimation network [17] to demonstrate that contextual cues can be transferred between different sets of human joints through transfer learning to improve convergence and performance.

We perform domain transfer with the help of transfer learning and compare it with frozen weights and random initialization settings. Concretely, for domain transfer, a pose estimation network is first trained on a subset of total joints, after which, a second, bigger network is trained on a different subset of joints, half of which is initialized by the weights of the previous network. Two more settings are investigated: random weight initialization and frozen weights. Random weight initialization is done by initializing all the weights of the second, bigger network randomly. In the frozen weights setting, the second, bigger network is trained after freezing/not updating the weights of the first network. The three settings are illustrated in Fig. 1. We compare the three settings with four different subset splits of human joints (Fig. 2) of the MPII dataset [1] and demonstrate that the transfer learning setting results in improved generalization performance and faster convergence.

The paper begins by detailing background information related to domain transfer, pose estimation, and the basis for our approach are discussed briefly in Section 2. The Methodology, Section 3, describes the pose estimation network employed for the experiments, the experimental settings and training details. The results and discussion forms Section 4 with Section 5 concluding the paper.

2 Background

Domain transfer utilizes information in one domain to help learn tasks in another domain. The two domains involved may represent separate datasets [7], classes in the same dataset [26] or different data modalities [9, 11, 28]. Effectiveness of domain transfer techniques has been tested in various problems like image classification [26], object detection [12] and semantic segmentation [5]. Transfer learning [26] and knowledge distillation [10, 27] are two popular ways of performing domain transfer. Yosinki et al. [26] demonstrate that performing transfer learning by initializing first $n$ layers of a base network with weights learned on approximately half of Imagenet [8] classes with the remaining layers randomly initialized improves generalization performance on the other half of Imagenet classes. Hinton et al. [10] introduced knowledge distillation by producing soft probability distribution over targets by modifying the softmax function and introducing an objective function consisting of those soft targets to train a student network. Techniques such as using weight regularizers to make weights of source and target domain networks similar [19] and adversarial learning [24, 16] are also employed for domain transfer.

With the advent of deep networks, there has been a significant progress in the field of human pose estimation. Toshev et al. [21] was among one of the earliest works incorporating deep neural networks (DNN) for pose estimation. Heatmap based pose estimation [17, 22, 6, 3] is the most widely used pose estimation technique. In heatmap based methods, joint heatmaps, equal to the number of joint locations present in the images are generated. Each heatmap represents a two dimensional probability distribution where each heatmap pixel represents the probability with which a joint is present in a particular pixel location. Intermediate supervision is commonly used in heatmap based methods, wherein loss is calculated at subsequent stages of the pose estimation network to refine heatmap predictions. Regression based approaches [21, 2, 4] are also prevalent in human pose estimation literature, however, their limitation is that the regression function is often sub-optimal. The work presented in this paper makes use of a heatmap based approach [17].

Domain transfer between separate keypoints subsets in deep human pose estimation is an unexplored avenue. Multimodal pose transfer methods exist in pose estimation literature. Zhao et al. [28] predict human pose from RF signals by transferring visual knowledge from an RGB images based pose estimation model [3] in a multimodal setting. Yang et al. [24] perform adversarial learning to transfer knowledge between annotated 3D human pose datasets and 2D in-the-wild images. Zhou et al. [29] use a weakly supervised approach and propose a geometric constrained to regularize depth predictions from 2D in-the-wild images . Zhang et al. [27] use knowledge distillation [10] to transfer knowledge from a teacher pose estimation network to a smaller network. In contrast, we are concerned with the task of domain transfer between separate body keypoint/joint subsets. Human body joints possess information about the location of one another through various constraints imposed by the overall body pose and structure. We present an approach to transfer rich contextual information between human body joints. Our work differs from approaches such as [26] in the way that instead of splitting a dataset based on classes, we have used subsets of human joint locations as separate domains and demonstrate that contextual knowledge can be transferred from one domain to the other using transfer learning. Our approach can be readily extended to other keypoint estimation problems such as facial landmark detection and 3D human pose estimation.

3 Methodology

In this section, the stacked hourglass network, experimental approach and training details are discussed. The MPII [1] dataset is used in the experiments which consists of around 28k training images and 11k testing images annotated with 16 body joints. Since the experiments involve evaluation on subsets of 16 annotated joints, a validation set of 3000 images is used for evaluation since the test annotations are not public.

3.1 Stacked Hourglass Network

Being the backbone of many state-of-the-art pose estimation algorithms [14, 23] on the MPII dataset [1], the stacked hourglass network [17] is used in our experiments. The hourglass architecture consists of repeated bottom-up and top-down processing in order to utilize features at various scales. Convolution and max pooling layers bring down the input resolution from $64\times 64$ pixels to $4\times 4$ pixels. After downsampling to a resolution of $4\times 4$ pixels, the features are upsampled with nearest neighbour upsampling and are combined with features of the same resolution. Several hourglass units are stacked together such that the output of one hourglass unit serves an input to the next hourglass unit. Intermediate supervision is applied such that mean squared loss is evaluated between the predicted heatmaps and ground truth heatmaps and gradients are back-propagated at every hourglass unit. The output of the network is a set of heatmaps equal to the number of joints with each pixel in the heatmap representing the probability with which the joint is present at that point.

3.2 Experiments

Domain transfer in keypoint estimation in performed by splitting the dataset joints into two subsets, $S_{1}$ and $S_{2}$ containing the same number of joints/keypoints. Both the subsets represent two separate domains. Note that $S_{1}\cap S_{2}$ may or may not be an empty set, such that the two domains differ in atleast one keypoint location. Domain transfer is experimentally determined using three different configurations illustrated in Fig. 1. The network performance is evaluated on the joint subset $S_{2}$ using the three experiment configurations discussed below:

3.2.1 Transfer learning

Since we are interested in determining the domain transfer from domain to another, the two-stack hourglass network trained on the subset $S_{1}$ of joints is jointly trained in conjugation with another two-stack hourglass network on the subset $S_{2}$ of joints (Fig. 1 (a)). The joint training is done such that supervision is performed for all the four hourglass units.

3.2.2 Frozen weights

This configuration is similar to transfer learning except that the weights of the first two-stack hourglass network trained on the subset $S_{1}$ of joints are frozen. Another two-stack hourglass network is trained on the remaining subset of eight joints $S_{2}$ such that the features obtained from the frozen network are used as an input for the network trained on subset $S_{2}$ (Fig. 1 (b)). In other words, the loss is calculated only for the last two hourglass units. This is done to avoid the possibility of mutual co-adaptation of weights between the two domains.

3.2.3 Random Initialization

A stacked hourglass network with four hourglass units with weights initialized randomly is trained on the subset $S_{2}$ of joints (Fig. 1 (c)).

The performance of the three configurations are evaluated with four different splits of subsets $S_{1}$ of joints and corresponding subset $S_{2}$ (split (a) -(d)) listed in the Table 1. Since, a very large number of subsets are possible, the joint subsets are chosen such that the result of knowledge transfer between adjacent limb joints when compared to random initialization can be evaluated as shown in Fig. 2. Note that all the three configurations have roughly the same number of parameters. The goal of the experiments is not to improve the state-of-the-art pose estimation benchmark, but to evaluate keypoint domain transfer.

3.3 Training details

The input image resolution for the network is $256\times 256$ pixels and the heatmap resolution is $64\times 64$ pixels. For all the experiments, rmsprop optimizer is used. The learning rate is divided by 5 each time the accuracy plateaus. Early stopping is implemented such that the model is said to be converged if there is no improvement in validation accuracy in 10 epochs with each epoch consisting of 8000 iterations. Data augmentation is carried out with $.75-1.25$ scale augmentation and $+/-30$ degree of rotation augmentation. The training is carried on an NVIDIA Geforce Titan X GPU.

4 Results

The PCKh metric [1] is used to evaluate pose estimation performance. A joint is correctly predicted if the distance between the ground truth and predicted joint location is less than half the length of the head segment. The PCKh values for the joint subset $S_{2}$ corresponding to the four joint subset splits with respect to the three experiment configurations i.e., transfer learning, frozen weights and random weight initialization is shown in Tables 2 - 5 respectively. Note that the pelvis and thorax torso joints are not considered in average PCKh computation since, being at the centre of the body, they are almost perfectly localized in all scenarios and just increase the average values. Figure 3 show the validation accuracy vs epochs curves comparisons between random initialization and transfer learning configurations. Apparently, transfer learning from pre-learned weights from other joints not only helps in achieving better accuracy values but also results in much faster convergence when compared to random initialization. The accuracy values shown in the validation accuracy vs epochs plots (Fig. 3) use the PCK metric [25].

4.1 Discussion

A number of interesting observations can be made:

Firstly, it can be observed from the splits (a) - (d) that features transferred from frozen weights of one domain i.e., subset $S_{1}$ do not achieve good accuracy on the second domain i.e subset $S_{2}$, when compared to random initialization and transfer learning. This is because, in the frozen weights configuration, since the loss is not computed on the first two hourglass units, there is no mutual co-adaptation of weights between the two domains. 2. 2.

From Table 2, in the case of split (a), it can be seen that transfer learning from torso joints improves accuracy on the wrists when compared to random initialization (83.9 vs 84.2), but does not have a considerable impact on the other limb joints. However, as elbows (split (b)) and ankles (split (c)) are included in subset $S_{1}$, we find that the performance on joints such as wrist (Table 3 and 4) and ankle (Table 4) becomes much better, such that, in split (c) the average PCKh over all joints (83.7) becomes better than random initialization (83.2). The effectiveness of domain transfer is further demonstrated in split (d) where wrists and ankles present in domain $S_{1}$ provide contextual knowledge to domain $S_{2}$. From Table 5, a higher average PCKh value (89.8) for transfer learning demonstrates the success of keypoint-wise domain transfer. The weights learned from one domain co-adapt with the other domain. 3. 3.

From Fig. 3 it is observed that the convergence and performance is complementary; i.e., in case of split (a) where, domain transfer does not result in any significant accuracy improvement, the convergence between random initialization and transfer learning configurations is comparable (Fig. 3 a). On the other hand, in case of split (d), where domain transfer performs better than random initialization, the convergence of transfer learning case is much better when compared to random initialization (Fig. 3 d). Whereas, in the two other cases the convergence is ”between” the two extreme situations of split (a) and (d). This is the result of the mutual co-adaptation of the network weights in learning adjacent limb joints. This shows that the weights learned on the one domain helps in better initialization of the cost function of the other domain which leads to faster convergence.

5 Conclusion

This paper introduces the KPTransfer approach for evaluating keypoint subset-wise domain transfer. We demonstrate that knowledge can be transferred between keypoint subsets in pose estimation such that the contextual cues present across domains helps in better generalization and faster convergence. This work also opens the door for cross-dataset domain transfer in keypoint estimation. Future work include: (1) Determining the exact joint subsets/domain between which domain transfer is most effective. (2) Extending this work to other problems like facial landmark detection and 3D human pose estimation. (3) Evaluation of the proposed keypoint domain transfer strategy with other pose estimation networks.