Joint Representation of Multiple Geometric Priors via a Shape Decomposition Model for Single Monocular 3D Pose Estimation

TL;DR

This paper introduces an unsupervised shape decomposition model for single monocular 3D human pose estimation, effectively handling depth ambiguity and complex deformations with limited training data.

Contribution

The paper proposes a novel Shape Decomposition Model and joint dictionary learning algorithm, enabling accurate 3D pose estimation from limited data and complex poses.

Findings

Outperforms existing methods on multiple datasets

Achieves significant improvements on complex deformation categories

Effective in in-the-wild image scenarios

Abstract

In this paper, we aim to recover the 3D human pose from 2D body joints of a single image. The major challenge in this task is the depth ambiguity since different 3D poses may produce similar 2D poses. Although many recent advances in this problem are found in both unsupervised and supervised learning approaches, the performances of most of these approaches are greatly affected by insufficient diversities and richness of training data. To alleviate this issue, we propose an unsupervised learning approach, which is capable of estimating various complex poses well under limited available training data. Specifically, we propose a Shape Decomposition Model (SDM) in which a 3D pose is considered as the superposition of two parts which are global structure together with some deformations. Based on SDM, we estimate these two parts explicitly by solving two sets of different distributed combination coefficients of geometric priors. In addition, to obtain geometric priors, a joint dictionary learning algorithm is proposed to extract both coarse and fine pose clues simultaneously from limited training data. Quantitative evaluations on several widely used datasets demonstrate that our approach yields better performances over other competitive approaches. Especially, on some categories with more complex deformations, significant improvements are achieved by our approach. Furthermore, qualitative experiments conducted on in-the-wild images also show the effectiveness of the proposed approach.

Figures

Tables

Table 1. Table 1: The result of per joint 3D error (mm) on Human3.6m dataset.

Methods	Direct.	Discuss	Eat	Greet	Phone	Photo	Pose	Buy	Sit	SitDown	Smoke	Wait	WalkDog	Walk	WalkPair	Avg
The supervised learning approaches
Tekin et al. [51]	102.4	147.2	88.8	125.3	118.0	182.7	112.4	129.2	138.9	224.9	118.4	138.8	126.3	55.1	65.8	125.0
Zhou et al. [52]	91.8	102.4	96.7	98.8	113.4	125.2	90.0	93.8	132.2	159.0	107.0	94.4	126.0	79.0	99.0	107.3
Tekin et al. [56]	85.0	108.8	84.4	98.9	119.4	95.7	98.5	93.8	73.8	170.4	85.1	117.0	113.7	62.1	94.8	100.1
Nie et al. [57]	90.1	88.2	85.7	95.6	103.9	103.0	92.4	90.4	117.9	136.4	98.5	94.4	90.6	86.0	89.5	97.5
Tome et al. [58]	65.0	73.5	76.8	86.4	86.3	110.7	68.9	74.8	110.2	173.9	85.0	85.8	86.3	71.4	73.1	88.4
Rogez et al. [59]	76.2	80.2	75.8	83.3	92.2	105.7	79.0	71.7	105.9	127.1	88.0	83.7	86.6	64.9	84.0	87.7
Pavlakos et al. [31]	67.4	71.9	66.7	69.1	72.0	77.0	65.0	68.3	83.7	96.5	71.7	65.8	74.9	59.1	63.2	71.9
Zhou et al. [60]	54.8	60.7	58.2	71.4	62.0	65.5	53.8	55.6	75.2	111.6	64.2	66.1	51.4	63.2	55.3	64.9
The unsupervised learning approach
Ionescu et al. [18]	132.7	183.6	132.4	164.4	162.1	205.9	150.6	171.3	151.6	243.0	162.1	170.7	177.1	96.6	127.9	162.1
Yu et al. [42]	85.1	112.7	104.9	122.1	139.1	135.9	105.9	166.2	117.5	226.9	120.0	117.7	137.4	99.3	106.5	126.5
Chen et al. [61]	89.9	97.6	90.0	107.9	107.3	139.2	93.6	136.1	133.1	240.1	106.7	106.2	114.1	87.0	90.6	114.2
Ours	103.6	101.7	83.6	106.3	86.1	109.3	96.1	114.5	97.0	138.3	83.7	93.7	98.4	100.0	112.0	99.5
Ours+refined	93.1	85.4	72.4	95.6	80.3	100.2	90.5	90.3	74.0	114.0	76.0	87.5	86.7	87.8	97.6	87.1

Table 2. Table 2: The result of mean estimation errors (mm) on Human3.6M.

Methods	Direct.	Discuss	Eat	Greet	Phone	Photo	Pose	Buy	Sit	SitDown	Smoke	Wait	WalkDog	Walk	WalkPair	Avg
The supervised learning approach
Lin et al. [62]	58.0	68.2	63.3	65.8	75.3	93.1	61.2	65.7	98.7	127.7	70.4	68.2	72.9	50.6	57.7	73.1
Morenonoguer et al. [30]	66.1	61.7	84.5	73.7	65.2	67.2	60.9	67.3	103.5	74.6	92.6	69.6	71.5	78.0	73.2	74.0
Pavlakos et al. [31]	47.5	50.5	48.3	49.3	50.7	55.2	46.1	48.0	61.1	78.1	51.1	48.3	52.9	41.5	46.4	51.9
Martinez et al. [32]	39.5	43.2	46.4	47.0	51.0	56.0	41.4	40.6	56.5	69.4	49.2	45.0	49.5	38.0	43.1	47.7
The unsupervised learning approach
Akhter et al. [63] $†$	199.2	177.6	161.8	197.8	176.2	186.5	195.4	167.3	160.7	173.7	177.8	181.9	176.2	198.6	192.7	181.1
Ramakrishna et al. [10] $†$	137.4	149.3	141.6	154.3	157.7	158.9	141.8	158.1	168.6	175.6	160.4	161.7	150.0	174.8	150.2	157.3
Zhou et al. [12]	87.4	109.3	87.1	103.2	116.2	143.3	106.9	99.8	124.5	199.2	107.4	118.1	114.2	79.4	97.7	113.0
Zhou et al. [11] $†$ [11]	99.7	95.8	87.9	116.8	108.3	107.3	93.5	95.3	109.1	137.5	106.0	102.2	106.5	110.4	115.2	106.7
Sanzari et al. [37]	48.8	56.3	96.0	84.8	96.5	105.6	66.3	107.4	116.9	129.6	97.8	65.9	130.5	92.6	102.2	93.2
Bogo eta l. [13]	62.0	60.2	67.8	76.5	92.1	77.0	73.0	75.3	100.3	137.3	83.4	77.3	86.8	79.7	87.7	82.3
Wang et al. [64]	50.0	60.0	54.7	56.6	65.7	78.6	52.7	54.8	85.9	118.0	62.5	59.5	64.9	42.0	49.5	63.8
Ours	54.0	49.5	43.1	53.0	40.4	55.1	54.1	57.2	40.1	54.5	39.0	48.7	45.7	46.1	53.2	47.7
Ours+refined	52.7	45.4	35.8	51.4	38.8	53.1	53.6	48.0	29.0	43.2	35.6	47.2	43.0	45.1	50.3	43.6

Table 3. Table 3: The result of mean estimation errors (mm) on HumanEva-I dataset.

Method	Walking(C1)			Jogging(C1)
	S1	S2	S3	S1	S2	S3	Avg
The supervised learning approach
Simo-Serra et al. [36]	65.1	48.6	73.5	74.2	46.6	32.2	56.7
Kostrikov et al. [40]	44.0	30.9	41.7	57.2	35.0	33.3	40.3
Bo et al. [4]	46.4	30.3	64.9	64.5	48.0	38.2	48.7
Katircioglu et al. [9]	29.3	17.9	59.5	-	-	-	35.6
Martinez et al. [32]	19.7	17.4	46.8	26.9	18.2	18.6	24.6
Pavlakos et al. [31]	22.3	19.5	29.7	28.9	21.9	23.8	24.3
Morenonoguer et al. [30]	19.7	13.0	24.9	39.7	20.0	21.0	23.1
The unsupervised learning approach
Radwan et al. [53]	75.1	99.8	93.8	79.2	89.8	99.4	89.5
Wang et al. [45]	71.9	75.7	85.3	62.6	77.7	54.4	71.3
Yasin et al. [54]	35.8	32.4	41.6	46.6	41.4	35.4	38.9
Wang et al. [20]	40.3	37.6	37.4	39.7	36.2	38.4	38.3
Zhou et al. [19]	34.3	31.6	49.3	48.6	34.0	30.0	37.9
Ours	34.5	30.7	32.8	32.3	33.7	32.4	32.7
Ours+refined	29.4	29.2	30.2	32.7	32.2	29.9	30.6

Table 4. Table 4: The mean estimation errors (mm) of different joints on CMU MoCap

Methods	Pelvis	RH	RK	RA	LH	LK	LA	Neck	Head	PS	RE	RW	LS	LE	LW	Avg
Ramakrishna et al. [10]	40.0	48.4	59.8	69.5	48.2	61.9	74.9	29.0	46.6	41.6	56.6	89.8	46.8	64.3	88.8	57.8
Zhou et al. [11]	35.3	45.5	53.3	64.2	48.3	53.8	68.1	25.6	44.5	39.9	58.0	76.7	43.7	61.0	75.7	52.9
Ours	33.1	42.8	49.2	59.1	41.3	48.8	67.4	23.2	36.2	32.7	51.3	68.7	37.4	49.3	68.4	47.3
Ours+refined	30.5	39.2	45.9	57.7	38.4	47.5	64.7	23.1	34.6	30.9	47.9	68.7	35.8	46.8	65.3	45.1

Equations

$$\mathbf{X}=\mathbf{S}\mathbf{R}\mathbf{Y}+\mathbf{T},$$

$$\mathbf{Y}={\mathbf{B}}^{\ast }\mathbf{c},$$

$$\underset{\mathbf{R},\mathbf{c},\mathbf{T}}{\min }\Vert \mathbf{c}{\Vert }_1\text{s.t.}\mathbf{X}=\mathbf{SR}({\mathbf{B}}^{\ast }\mathbf{c})+\mathbf{T},$$

$$\mathbf{Y}=\mathbf{U}+\mathbf{V},$$

$$\mathbf{U}={\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}},$$

$$\mathbf{V}={\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}},$$

$$\underset{\mathbf{R},{\mathbf{c}}_{\mathbf{u}},{\mathbf{c}}_{\mathbf{v}}}{\min }\varphi ({\mathbf{c}}_{\mathbf{u}})+\phi ({\mathbf{c}}_{\mathbf{v}})\text{s.t.}\mathbf{X}=(\mathbf{SR})({\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}}+{\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}})+\mathbf{T},$$

$$\underset{\mathbf{R},{\mathbf{c}}_{\mathbf{u}},{\mathbf{c}}_{\mathbf{v}}}{\min }\frac{1}{2}\Vert \mathbf{X}-(\mathbf{SR})({\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}}+{\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}})+\mathbf{T}{\Vert }_2^2+\alpha \Vert {\mathbf{c}}_{\mathbf{u}}{\Vert }_1+\beta \Vert {\mathbf{c}}_{\mathbf{v}}{\Vert }_2^2,$$

$$\underset{{\mathbf{R}}^{\ast },{\mathbf{c}}_{\mathbf{u}},{\mathbf{c}}_{\mathbf{v}}}{\min }\frac{1}{2}\Vert \mathbf{X}-{\mathbf{R}}^{\ast }({\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}}+{\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}}){\Vert }_2^2+\alpha \Vert {\mathbf{c}}_{\mathbf{u}}{\Vert }_1+\beta \Vert {\mathbf{c}}_{\mathbf{v}}{\Vert }_2^2,$$

$${\mathbf{R}}^{\ast }=\underset{{\mathbf{R}}^{\ast }}{\min }\frac{1}{2}\Vert \mathbf{X}-{\mathbf{R}}^{\ast }({\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}}+{\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}}){\Vert }_2^2.$$

$${\mathbf{c}}_{\mathbf{u}}$$

$${\mathbf{c}}_v$$

$${\mathbf{c}}_{\mathbf{v}}=({\mathbf{Z}}^T\mathbf{Z}+\beta \text{I}{)}^{-1}{\mathbf{Z}}^T(\mathbf{X}-{\mathbf{R}}^{\ast }({\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}})).$$

$${\mathbf{D}}_m$$

$$={\mathbf{B}}_{\mathbf{u}}^{\ast }{\mathbf{c}}_{\mathbf{u}m}+{\mathbf{B}}_{\mathbf{v}}^{\ast }{\mathbf{c}}_{\mathbf{v}m},$$

$$\underset{\{{\mathbf{B}}_{\mathbf{u}},{\mathbf{B}}_{\mathbf{v}}{\}}_1^k,{\mathbf{C}}_{\mathbf{u}},{\mathbf{C}}_{\mathbf{v}}}{\min }\sum _{m=1}^N\frac{1}{2}\Vert {\mathbf{D}}_m-\sum _{j=1}^k({\mathbf{B}}_{\mathbf{u}j}{c}_{ujm}+{\mathbf{B}}_{\mathbf{v}j}{c}_{vjm}){\Vert }_2^2+\gamma \Vert {\mathbf{C}}_{\mathbf{u}}{\Vert }_1+\eta \Vert {\mathbf{C}}_{\mathbf{v}}{\Vert }_2^2,$$

$$\mathbf{er}{\mathbf{r}}_{\mathbf{p}}(\hat{\mathbf{Y}},{\mathbf{Y}}_g)=\frac{1}{N}\sum _{i=1}^N(\Vert {\hat{\mathbf{y}}}_i-{\mathbf{y}}_i{\Vert }_2),$$

$$\mathcal{L}({\mathbf{B}}_{\mathbf{u}}^{\ast },{\mathbf{B}}_{\mathbf{v}}^{\ast },{\mathbf{C}}_{\mathbf{u}},{\mathbf{C}}_{\mathbf{v}})=\frac{1}{2}\Vert {\mathbf{D}}_m-\sum _{j=1}^k({\mathbf{B}}_{\mathbf{u}j}{c}_{ujm}+{\mathbf{B}}_{\mathbf{v}j}{c}_{vjm}){\Vert }_2^2+\gamma \Vert {\mathbf{C}}_{\mathbf{u}}{\Vert }_1+\eta \Vert {\mathbf{C}}_{\mathbf{v}}{\Vert }_2^2,$$

Full text

††journal: Journal of LaTeX Templates

1 Introduction

In many computer vision applications, such as human-robot interaction, human action surveillance, and virtual reality (VR), etc, it is necessary to recover the 3D human pose from 2D images. However, this is a challenging task due to the lack of depth information in 2D images. To overcome this difficulty, some literature have presented typical solutions by assuming that the RGB-D image [1] or the multi-view image [2, 3] is available. Benefited from these supplementary data, many works have made impressive progress over the past years. However, in many real-life scenes, it is infeasible to develop specific hardware equipment for the capture of depth information or multi-view images. Thus, the ultimate goal is to recover the 3D human pose from a single RGB image which can be obtained easily.

Lifting 2D observations of a single image to 3D space, there are two main factors leading to difficulties. Firstly, influenced by many factors, such as illuminations, cluttered backgrounds or occlusions, 2D human features are hard to locate in the image. Secondly, even ground-truth 2D body features are given, it is still a highly ill-posed problem since similar 2D projections may be projected from different 3D poses. To overcome the above difficulties, existing approaches have proposed different solutions over the past years.

The first class of approaches directly learn a mapping from image features to 3D pose [4] or search the best-matched example from an abundant of stored correspondences between 2D observations and its 3D pose [5, 6]. Benefited from the development of Convolutional Networks (ConvNets), researchers focus on the ConvNets-based of end-to-end learning approaches to estimate the 3D pose from the 2D image features [7, 8, 9]. Another category is ed as the model-based method [10, 11], which infers the 3D pose from 2D joints of an image by optimizing a designed objective function based on a given human body model. Recently, the 3D pose estimation problem has been addressed by a hybrid approaches [12, 13]. In these works, 2D joints are used as human body features which are detected by existing robust 2D detectors at first [14]. Then, the model-based approaches are applied as a post-processing optimization to infer the 3D pose from detected 2D joints. Since the 2D joints are relatively easy to obtain, the model-based approaches become critical to the performance of the 3D pose estimation.

Sparse representation (SR) model [15] is an effective model-based method to combine 2D annotations with 3D priors learned from existing Mocap dataset [16, 17, 18]. Based on SR model, each 3D pose is represented as a linear combination of a few of pre-defined 3D bases. The effectiveness of the sparsity combination assumption has been validated in many SR based approaches [11, 19, 20, 21, 22, 23]. However, the success of SR based algorithms highly depend on the assumption that sufficient representative 3D bases must be guaranteed for every input signal. For example, we consider an extreme case that the training data contain only “Walk” action. In this case, the standard SR based model [10] fails to reconstruct the deformations of a “Sit” pose, as the 3D pose with the color blue shown in Figure 1.

The main reason for this issue is that 3D bases obtained from “Walk” training data are not representative enough to approximate the input “Sit” pose. In addition, the sparsity assumption limits SR model to capture more implicit details from all bases. Thus, the diversities of 3D training data are needed for SR based approaches to handle various complex poses. However, rich and diverse training samples are difficult to obtain since it is almost infeasible to collect the 3D annotations data in unconstrained scenes. In addition, although several available Motion Capture (MoCap) datasets are widely used as training sources, they are collected in indoor. Such a collection environment greatly limits the diversity of subjects and actions. In order to augment the training data, some researchers propose different synthesize techniques [24]. However, the same distributions between these synthesized data cannot be guaranteed. As a result, the diversity requirement of training data for SR based approaches is hard to be satisfied.

In this paper, in order to enhance the standard SR model to estimate various complex poses under available training data, we propose the following solutions. First of all, inspired by the fact that a complex deformation human pose is usually derived from a standard pose, we propose a shape decomposition model (SDM) in which a 3D pose is separated into two parts which are the global structure and the corresponding deformations. It is a natural assumption since most of the deformed poses can be considered as a standard pose added with some specific variations. The advantage of SDM is that the complex deformations of the 3D pose are explicitly modeled for the subsequent refined estimation.

Based on SDM, for the estimation of the global structure of 3D poses, similar as the sparse assumption used in most SR based approaches, we represent the global structure as a sparse linear combination of global bases. While for deformations of 3D poses, we model pose deformations as a dense liner combination of deformation bases instead of the sparse combination. The main reason for this strategy is that different pose may have similar bends in some part of the human body, for example, the squatting and weightlifting have a similar knee bending. The dense constraint means that each 3D pose is estimated by using deformation clues from all 3D bases which may from other types of 3D training poses.

Further, to learn global structure and deformation dictionaries which contain a great deal of 3D bases, we propose joint learning of multiple geometric priors approach. Comparing with the single dictionary learning in most of SR based approaches, the proposed learning approach is able to obtain the dual dictionaries which captures more implicit information from the limited training data. The overview of the proposed approach is depicted in Figure 2. Concretely, we train two sets of basis from MoCap dataset in advance. Then, a 3D pose is obtained by adding results from two sets of linear combinations of basis. Next, two sets of combined coefficients are solving by minimizing the given 2D pose of an image and the projected 2D pose (projected from the estimated 3D pose).

Our strategies are able to reconstruct a reasonable 3D pose even if the type of input pose is not included in the training data, as the 3D pose with the color yellow shown in the Figure 1. In general, the main contributions of this paper are summarized as follows:

Considering that a human pose can be separated as a global structure and corresponding deformations, we propose a Shape Decomposition Model (SDM) which allows us to estimate two parts of 3D pose explicitly.

• 2.

A hybrid representation strategy is presented to model two parts of the 3D pose. Specially, for global structure and deformations parts, we represent them as a sparse and dense linear combination of learned 3D bases respectively. In this collaborative representation, a complex 3D pose is estimated from coarse to fine.

• 3.

To learn geometric priori, we propose the joint learning approach of multiple geometric priors, in which two over-complete dictionaries, including a number of 3D bases, are learned simultaneously. Our dictionary learning approach extract more implicit information from the available limited training data.

The remainder of this paper is organized as follows: the related works are reviewed in Section 2. The problem formulation is described in Section 3. The proposed model and inference details are given in Section 4. Section 4.3 presents the joint learning approach of dual-dictionaries. Section 5 describes experimental results and analysis. Section 6 concludes the paper.

2 Related Work

It is well known that estimating the 3D pose form a single RGB image is a very ambiguous problem. This issue can be alleviated by many solutions which can be roughly classified as the model-free and model-based methods. More recently, hybrid approaches have emerged by combining the above two classes.

The model-free methods estimate the 3D pose from 2D observations of the image directly. Elgammal and Lee  [25] infer the 3D pose from human silhouettes by learning the representations of activity manifolds. Ankur and Bill [26]apply a nonlinear regression to estimate pose from shape descriptor vectors of silhouettes. To improve the estimation performance, Sedai al. [27] combines shape and appearance descriptors to infer the 3D pose jointly based on a discriminative learning framework. Considering that the above approaches cannot ensure the interdependencies between outputs, Bo and Sminchisescu [4] use the Kullback-Leibler divergence to match the distributions of inputs and outputs. Unlike the above learning-based approaches, Jiang al. [5] searched the optimal poses through millions of exemplars using the nearest neighbor scheme. As the deep convolutional networks yield significant performance in many areas, various ConvNets architectures are designed to estimate the 3D pose [28, 7, 29, 30, 31, 32, 33]. However, the collection of a large of paired 2D-3D correspondence for the performance of model-free techniques is still a challenge although many data augmentation approaches have been proposed [24, 34].

Unlike the model-free approaches, the model-based method [35, 10, 36, 11, 13, 37, 38, 19, 20] only use 3D annotations to fit their models. Based on prior knowledge, the model-based methods mainly contains two parts, the modeling and the inferring. A commonly used model is the active shape model (ASM) [39], where a 3D human pose is represented as a linear combination of 3D bases [10, 35, 11, 19, 20]. Ramakrishna et al. [10] estimate the model parameters by minimizing the projection error based on sparse representation model. Simo-Serra et al. [35] propagate the noise to the pose space by using a stochastic sampling strategy. Zhou et al. [11] propose a convex formulation to solve the non-convex problems of the orthogonality constraint imposed on the objective function. Instead of using the $\ell_{2}$-norm to measure the reconstruction error, Want et al. [20] apply the $\ell_{1}$-norm to minimize the projection error to improve the estimation robustness. Apart from ASM, Kostrikov et al. [40] apply 3D pictorial structure to estimate the 3D pose based on regression forests. Based on the mixture of pictorial structures, Radwan et al. [41] impose kinematic and orientation constraints on the reasoning model for the self-occlusion pose estimation.

The hybrid methods combine the model-free and model-based approaches to estimate 3D human pose from an image. In the strategy, the existing generative 2D detectors [14] are only used to detect 2D features, such as 2D body joints, and then the generative approaches are applied to infer 3D pose from these features [36, 42, 13, 19]. Since the literature worked on the 3D pose estimation are vast, we are unable to cover all approaches in this section, the readers interested in this topic are referred to the survey [43].

3 Problem Formulation

3.1 Dependence between the 2D and 3D poses

In this paper, a human body is represented as a skeleton with $P$ joints, its 2D pose and 3D pose are described as $\bm{X}=\{\bm{j^{\prime}}_{i}\}_{i=1}^{P}\in\mathbb{R}^{2\times P}$ and $\bm{Y}=\{\bm{j}_{i}\}_{i=1}^{P}\in\mathbb{R}^{3\times P}$ respectively, where $j^{\prime}\in\mathbb{R}^{2}$ and $j\in\mathbb{R}^{3}$ denote joint coordinates in 2D and 3D spaces respectively. Based on the assumption of a weak perspective camera model, the projection relationship between the 2D pose and its corresponding 3D pose is modeled as

[TABLE]

where $\bm{R}\in\mathbb{R}^{3\times 3}$ represents the rotation matrix and $\bm{T}=\bm{t}\bm{1}^{\top}\in\mathbb{R}^{2\times P}$ represents the translation vector, where $\bm{t}\in\mathbb{R}^{2}$. \bm{S}=\left[\begin{array}[]{ccc}s_{x}&0&0\\ 0&s_{y}&0\\ \end{array}\right] denotes the camera calibration matrix, where $s_{x}$ and $s_{y}$ are scaling factors in the x-axis and y-axis respectively. Given a 2D pose, the aiming of this paper is to find the 3D pose $Y$ whose projection is as equal as possible to the given 2D pose $X$. However, this is a ill-posed problem since there may be many 3D poses that satisfies Eq. (1). Fortunately, this problem can be alleviated by sparse representation (SR) model [15].

3.2 Sparse Representation for 3D pose estimation

Based on SR model [15], a 3D pose is represented as a sparse linear combination of a set of bases as follow:

[TABLE]

where $\bm{B^{*}}=\{\bm{B}_{j}\}_{j=1}^{k}$ is the over-completed dictionary, $\bm{B}_{j}\in\mathbb{R}^{3\times P}$ represents $j$th 3D basis pose, $k$ is the size of dictionary. $\bm{c}=[c_{1},...c_{k}]^{T}$ is the sparse coefficient vector in which most of entries are expected to be zero. As a result, only a few bases in dictionary are activated to express the 3D pose.

Since the dictionary is pre-learned from the training data, the problem of estimating the 3D pose $Y$ is converted to the solving of sparse coefficient $c_{i}$. By combining Eq. (1) and Eq. (2), the standard objective function based on SR model is defined as

[TABLE]

where $\|\cdot\|_{1}$ is $\ell_{1}$-norm which used to ensure the sparsity of coefficients vector $\bm{c}$. Notice that $\ell_{1}$-norm is the convex approximation of $\ell_{0}$-norm since the solving of $\ell_{0}$-norm is NP-hard [44].

Eq. (3) is the standard SR formulation to model the task of 3D pose estimation from 2D joints of an image, which is widely used in most unsupervised learning approaches [10, 11, 45]. The sparsity constraint $\|\cdot\|_{1}$ in Eq. (3) means that only a few bases poses are selected from the dictionary to represent a 3D pose. In addition, these activated bases are often in the same category as input 2D pose, and the more similar bases to the input signal are usually assigned higher combination weight values. This is the reason that SR model also performs well in the classification task [23].

However, we observed that two poses with large overall differences may have similar bends in some local joints the human body, thus those unselected basis pose also are able to provide useful clues for deformation estimations of the 3D pose. Unfortunately, the standard SR model fails to capture detail deformations from unselected bases which may learned from different category training data. As a result, the estimation performance of SR based approaches is affected, especially when the type of input pose is not included in training data. To alleviate this problem, we propose our approach in the coming section.

4 Proposed Models

4.1 Shape Decomposition Model and 3D Pose Inference

Considering that an arbitrary 3D pose $\bm{Y}$ contains both the global structure and its corresponding detail deformations, we describe a 3D pose as

[TABLE]

where $\bm{U}\in\mathbb{R}^{3\times P}$ and $\bm{V}\in\mathbb{R}^{3\times P}$ denotes the global structure of the 3D pose and its deformations respectively. Applying the active shape model [39], we represent each part as

[TABLE]

where $\bm{B_{u}^{*}}=\{\bm{B}_{\bm{u}j}\}_{j=1}^{k}$and $\bm{B_{v}^{*}}=\{\bm{B}_{\bm{v}j}\}_{j=1}^{k}$ denote the global structure and deformation dictionaries respectively, whose learning strategy is described in section 4.3. $\bm{c_{u}}\in\mathbb{R}^{k}$ and $\bm{c_{v}}\in\mathbb{R}^{k}$ are corresponding combination coefficient vectors. We term Eq. (4) as Shape Decomposition Model (SDM). Since the dictionaries $\bm{B_{u}^{*}},\bm{B_{v}^{*}}$ is pre-defined, the 3D pose can be calculated by using two sets of combination coefficients $\bm{c_{u}^{*}},\bm{c_{v}^{*}}$ which are possible to be solved by imposing suitable constraints. Thus, by combining Eq. (1) and Eq. (4)-(6), we formulate the optimization problem as

[TABLE]

where $\phi(\cdot)$ and $\varphi(\cdot)$ are constraint terms on the coding of global structures of 3D poses and its deformations information.

For the coding of global structure part, based on the assumption that the 3D pose lies in a small subspace [10, 11, 12, 20], we use $\|\cdot\|_{1}$ on $\bm{c_{u}}$ to enforce the solution sparsity. While for the coding of body deformations, considering that different types poses may have similar deformations in some local body part, we use the dense constraint $\|\cdot\|_{2}$ on $\bm{c_{v}}$ to produce a dense solution which implies that pose deformation is represented collaboratively by all bases in the dictionary $\bm{B^{*}_{v}}$. This strategy is successfully applied in the face recognition task [46]. In addition, considering that Gaussian noises may exist in Eq. (4), we relax the equality constraint of Eq. (7). As a result, Eq. (7) is rewritten as following Lagrangian form

[TABLE]

where $\alpha$ and $\beta$ are balance factors for each minimization term.

For further simplicity, we remove the translation vector $\bm{T}$ in  (8) by centralizing the data, and we introduce a new variable $\bm{R^{*}}\in\mathbb{R}^{2\times 3}$ to denote the product of $\bm{S}$ and $\bm{R}$. Ultimately, the final objective function is given as

[TABLE]

4.2 Optimization

To solve the optimization problem of the model (9), we apply the Alternating Direction Method of Multipliers (ADMM) [47], in which the variables $\bm{R^{*}},\bm{c_{u}},\bm{c_{v}}$ are updated alternatively.

Updating $\bm{R^{*}}$: We fix $\bm{c_{u}},\bm{c_{v}}$, $\bm{R^{*}}$ is obtain by minimizing the following subproblem:

[TABLE]

The problem (10) is a differentiable function which can be solved by the manifold optimization toolbox [48].

Updating $\bm{c_{u}}$: By fixing $\bm{R^{*}}$ and $\bm{c_{v}}$, the subproblem about $\bm{c_{u}}$ is given as

[TABLE]

The problem (11) is convex, thus it can be solved by $\ell_{1}$-norm minimization solvers, such as Accelerated Proximal Gradient (APG) algorithm [49].

Updating $\bm{c_{v}}$: Fixing $\bm{R^{*}}$ and $\bm{c_{u}}$, the $\bm{c_{v}}$ is solved by following optimization problem:

[TABLE]

The sub-problem (12) consists of two differentiable functions, it exists a closed-form solution since it is a convex issue. By introducing an auxiliary variable $\bm{Z}=\{\bm{R^{*}B}_{\bm{v}i}\}_{i=1}^{k}$, we present the solution of $\bm{c_{v}}$ as

[TABLE]

where I is identity matrix. The completed reconstruction approach based on SDM is summarized in Algorithm 1.

4.3 Dictionary Learning

The global structure dictionary $\bm{B}^{*}_{u}$ and the deformations dictionary $\bm{B}^{*}_{v}$ in Eq. (9) should be properly learned to provide powerful priors for the 3D pose estimation. Given a set of training set $\bm{D^{*}}=[\bm{D}_{1},...\bm{D}_{m},...\bm{D}_{N}]$, we apply SDM on each training pose $\bm{D}_{m}$ which is formulated as

[TABLE]

where $\bm{D}_{m}$ is the $m$-th 3D pose in training set. $\bm{U}_{m}$, $\bm{V}_{m}$ are the global structure and deformations of the $m$-th 3D training pose. $N$ is the size of training set. Similar to Eq. (9), by introducing the sparsity and density constraints, the object function of dictionary learning is presented as

[TABLE]

where $c_{ujm}$ and $c_{vjm}$ denote the $j$-th sparsity and dense coefficient vectors respectively for the $m$-th training pose. $\bm{C_{u}}=\{\bm{c}_{\bm{u}m}\}_{m=1}^{N}$ and $\bm{C_{v}}=\{\bm{c}_{\bm{v}m}\}_{m=1}^{N}$ are the combination matrices by stacking coefficient vectors $\bm{c_{u}}=[c_{u1},...,c_{uk}]^{\top}$ and $\bm{c_{v}}=[c_{v1},...,c_{vk}]^{\top}$ of all training poses respectively. $\gamma$ and $\eta$ are balance parameters for sparse and dense representation terms respectively. The formulation (14) can also be solved by ADMM scheme. Given training data $\bm{D^{*}}$, we alternately update the $\bm{B^{*}_{u}},\bm{C_{u}},\bm{B^{*}_{v}},\bm{C_{v}}$ while fixing the others. The algorithm is presented in Appendix A.

5 Experiments

In this section, we evaluate the proposed approach on the task of estimating the 3D pose from 2D joints of an image. In the following subsections, the experimental settings are presented in Section 5.1. The parameter sensitivity experiments are reported in Sections 5.2. Quantitative results and analysis conducted on different datasets are provided in Section 5.3, 5.4, and 5.5. Qualitative results are given in 5.6.

5.1 Experimental settings

5.1.1 Evaluation Dataset and Protocol

The proposed approach is evaluated on four publicly available datasets, they are Human3.6M [18], HumanEva-I [17], CMU Mocap [16], and MPII Human Pose [50]. Specially, the last one is used for the qualitative evaluation, and others for the quantitative evaluations. Notice that we only use the 3D pose date for training rather than paired 2D-3D data with images in most supervised approaches. [51, 52, 32]

Human3.6M: This is a large scale dataset that contains synchronized videos with corresponding 2D-3D human poses of 11 subjects. Each subject performs 15 scenarios, including sitting, smoking, and eating, etc. These processes are captured by an indoor MoCap system with 15 sensors (video and motion cameras, time-of-flight sensor). The evaluation data were downsampled from 50fps to 10fps. The data from subjects (S1, S5, S6, S7, S8) was used for training, and (S9, S11) for testing. This is the standard evaluation protocol which is applied in most approaches [12, 8, 29, 19].

HumanEva-I: Compared to Human3.6M dataset, HumanEva-I dataset provides the smaller synchronized sequences and corresponding 2D-3D annotations. Specifically, six actions, such as jogging, catching, boxing, etc., are performed under seven camera views by four subjects. Following the same evaluation protocols in most literature [53, 45, 36, 40, 4, 54, 19, 9], we evaluated our approach on categories of ”Walking” and ”Jogging” performed by subjects S1, S2, and S3 from ”validation” set.

CMU Mocap: This dataset contains 30 different actions performed by 144 subjects, and collects more than three million 3D human poses. For the evaluation, eight motions (basketball, dancing, boxing, jumping, running, climbing and walking) are collected from different subjects. In our evaluation, six sequences are collected from different motions respectively, where three sequences are used as training sets and the remaining three sequences are treated as test sets. To generate input 2D pose, we simulate a 360-degree rotating orthogonal camera to projected the 3D pose into the 2D space.

MPII: This dataset is used for the qualitative evaluation. In 2D pose estimation works, MPII is a widely used dataset which is consists of about 25k images and 40k 2D poses. These 2D poses are obtained from different camera viewpoints and exist occluded or truncated case. We select different motions and viewpoints to evaluate the effectiveness of the proposed approach.

5.1.2 Evaluation Metrics

By assuming that the 2D joints are given, the proposed approach recover the 3D pose from given 2D joints. Two evaluation metrics are considered in experiments. The first is per joint 3D error which measures the mean euclidean distance between the joints of the estimated 3D pose and the ground truth 3D pose. This metric is formulated as follow

[TABLE]

where $\bm{\hat{Y}}$ and $\bm{Y}_{g}$ represent the estimated 3D pose and the ground-truth 3D pose respectively. The second is 3D estimation error which also calculates the euclidean distance between two 3D poses but after a rigid alignment.

5.1.3 Implementation Details

The Algorithm 2 was used to learn the global structure and deformation dictionaries. We set both dictionaries size to $K=\{258,168,158\}$ for Human3.6M, HumanEva-I, and CMU Mocap respectively. The hyper-parameters $\gamma,\eta$ in Eq. (15) were fixed in $\{0.01,1\}$. For all evaluated dataset, notice that we use all training examples to train our dictionaries that does not distinguish specific actions.

For the 3D pose estimation, we set hyper-parameters $\alpha,\beta$ in the proposed model Eq. (9) to $\{\alpha=0.4,\beta=20\}$, $\{\alpha=0.1,\beta=10\}$, and $\{\alpha=1.5,\beta=20\}$ for Human3.6M, HumanEva-I, and CMU Mocap respectively. For these hyper-parameters, we perform ablation analyses in the coming subsection (5.2).

For model variables (rotation matrix $\bm{R}$ and combination coefficient vectors $\bm{c_{u}},\bm{c_{v}}$) in Eq. (9), we initialize them as an identity matrix, zero vectors. To evaluate the influence of initialization, we also use a convex relaxation algorithm [55] to initialize rotation matrix $\bm{R}$. In evaluation comparision, the proposed approach with a fine initialization is named as ”Our+refined”. In addition, in all experiments, the maximum iterations $L_{max}$ and the convergence tolerance $\tau$ were set to 10,000 and $10^{-6}$ respectively.

5.2 Parameter sensitivity

In the proposed model (9), hyper-parameters $\alpha$ and $\beta$ are used to balance the trade-off of each optimization term. To study the performance of our approach as these hyper-parameters change, we conduct experiments of our approach using different hyper-parameters $\alpha$ and $\beta$ on three datasets. For each dataset, we selected 100 examples randomly for the evaluation.

First of all, we perform hyper-parameter ablation analysis on parameter $\alpha$ that controls the sparsity of sparse representation. In experiments, the value of $\alpha$ is varied in the range [0, 5] while $\beta$ is fixed to 18, 10 and 8 on Human3.6m, HumanEva-I, and CMU Mocap datasets respectively. As shown in Figure 3(a), the estimation error decreases rapidly when $\alpha$ becomes nonzero. This indicates that the sparsity is important for reconstruction performance. As $\alpha$ increases, the performance becomes very stable across all datasets. However, we find that the curve of HumanEva-I has a valley when $\alpha$ is equal to $0.1$.

Further, the ablation analysis is also conducted on $\beta$ which handles the intensity of dense representation. The value of $\beta$ is varied in the range [0, 20] while $\alpha$ is fixed to 0.15, 0.1 and 1. Similar observation is made in the ablation experiment on $\beta$, as shown in Figure 3(b). As $\beta$ increases from zeros, the estimation errors begin to fall sharply and tend to stabilize. This indicates the importance of the dense constraint for the proposed model. In general, the solution of our model is not very sensitive to $\alpha$ and $\beta$ when the values are in the proper range.

5.3 Quantitative Evaluation on Human3.6M Dataset

The first comparative study is conducted on Human3.6M dataset. In this dataset, the per joint 3D error is widely used as the performance metric in most works, while the results of estimation error are also reported in some literature. Thus, for a fair comparison, the performance from both two metrics are evaluated respectively in this paper. To evaluate the effectiveness of our approach, we compare representative works of both unsupervised and supervised learning approaches in the literature, which are organized into two groups. The result from our approach after a refined initialization is also presented, named as ”Ours+refined”. All compared figures are obtained from original papers except for the works with the marker $\dagger$ that means the results are obtained from [13].

First of all, the performances of the proposed approaches and representative works are reported in Table 1 in terms of per joint 3D error. As seen in the table, compared to unsupervised learning works, our approach (Ours) achieves the best performance in most cases and more than 12% improvement on average. The mean accuracy is further boosted more than 23% when we use the better initialization. Even comparison to supervised learning works, our approaches are still comparable. Specially, our approach (Ours+refined) outperforms most supervised learning algorithms [51, 52, 56, 57, 58, 59] on average.

Moreover, experimental results of mean estimation error from our approaches and compared works are shown in Table 2. Similar to the observation on the per joint 3D error metric, there is a wide performance margin between the proposed approach and compared unsupervised learning approaches. Especially, our approach with refined initialization (Ours+refined) consistently outperforms all considered unsupervised and supervised learning approaches by more than 31% and 8% improvements on average respectively. In addition, the proposed approach without refined initialization (Ours) still achieves the best performance than all unsupervised and many supervised learning approaches [62, 30, 31] by more than 25% and 15% on average respectively.

In Table 1 and Table 2, we observed that our approach consistently performs better in categories with more deformations, such as “Buy”, “Sit”, and “SitDown”, etc. For example, on the motion of “Sit”, the best performance is always observed on the results from our approach. Especially, the the-state-of-arts estimation error on “Sit” is achieved by the proposed approach. However, some exceptions are also observed on categories, including “Direct.”, “Walk”, and “WalkPair”. In these cases, our approach leads the slightly worse performance. This may be due to that the over-fitting issue happens in the proposed approach since the testing poses included in these categories have less deformations.

5.4 Quantitative Evaluation on HumanEva-I Dataset

The comparative evaluation is also presented in HumanEva-I dataset. Similar to the evaluation of Human3.6M dataset, the quantitative results from both supervised and unsupervised learning approaches are considered, as reported in Table 3.

We clearly observed that our approach outperforms all unsupervised learning approaches by more than 13% on average. This figure is further increased to 19% when we use better initialization. In addition, even compared to supervised learning approaches, the performance of our approach is still comparable. Specially, the proposed approach outperforms some competitive works [36, 40, 4, 9] by more than 8% on average.

Although our approach without refined initialization outperforms other unsupervised learning methods in most categories, there are still a few exceptions. For example, the performances of our approach are slightly worse than the literature [19] on some categories, such as “Walking” of the subject the “S1” and “Jogging” of the subject “S3”. However, the accuracies on these cases are considerably improved when we use a better initialization. In addition, notice that the work [19] action specific dictionaries for each subject separately. However, for the model generality, we train our dictionaries without distinction of specific action or subjects.

In most cases, a fine initialization usually boosts the performance of our approach. However, there are also very few exceptions. As we observed in Table 3, the fine initialization (Ours+refined) leads to a slightly worse reconstruction in “Jogging” of subject “S1”. It may happen that the solution provided by the convex approach is not a good initialization.

5.5 Quantitative Evaluation on CMU MoCap Dataset

Compared to comprehensive evaluations on HumanEva-I and Human3.6m, there are fewer approaches to evaluate on CMU Mocap. However, we notice that still some generalization experiments in this dataset are reported on some literature [10, 11] which both are SR based approaches. Since these approaches are the most related works to our approach, we also evaluate on this dataset to show the effectiveness of the proposed approach.

To evaluate our model on different motions, we observed that the proposed approaches outperforms all compared baselines across all motions categories, as shown in Figure 4(a). Due to the smaller motion range and deformation of “Walk” category, all the compared algorithms have lower errors and no evident differences between them. However, in some more complex categories, such as “Dance” and “Hoop”, we can clearly see that our algorithms boost performance. In addition, we present the percentage of different error ranges, as shown in Figure 4(b). The proposed approaches perform better reconstruction on most testing examples than the baselines. Specially, the percentage of our approach (Ours+refined) exceeds 80% when the mean estimation errors are smaller than 60 (mm). However, this percentage is decreased to about 60% achieved by the [10] and about 70% by [11]. Finally, we evaluate the robustness of our approach against the Gaussian noises with various standard deviations, as shown in 4(c). The accuracies of our approach are consistently better than the compared approaches under all noise levels.

In addition, the mean estimation errors on different 15 body joints, including Head, Neck, Left Shoulder (LS), Right Shoulder (RS), etc., are shown in Table 4.

As seen from the table, our approaches consistently achieve lower estimation errors across all body joints than compared baselines. Specially, more than 10% (Ours) and 14% (Ours+refined) on average improvements are achieved by the proposed approaches. Since severe deformation and self-occlusion are more likely to occur in some joints, such as left wrist (LW), right wrist (RW), left ankle (LA), and right ankle (RA), the performances of all compared algorithms are usually worse on these joints. However, the proposed approach shows more evident improvements in these joints.

5.6 Qualitative Evaluation on MPII Dataset

The applicability of our approach for the 3D pose estimation on in-the-wild images is also demonstrated. To evaluate the effectiveness of the proposed approach, we selected eight various actions and challenging examples. The qualitative results are presented in Figure 5.

Notice that this dataset did not provide 3D annotations, thus we use the dictionary learned from CMU Mocap dataset for the 3D pose inference. Even so, we clearly observed that the proposed method recovers the 3D human poses successfully from various actions and subjects. However, we also represent some failure cases, as shown in Figure 6. Similar to most algorithms, the depth ambiguity and serious occlusions are still problems.

6 Conclusion

In this work, we aim to enhance the standard SR model to recover various complex poses under available limited training data. Firstly, we proposed the Shape Decomposition Model (SDM) in which a complex 3D pose is represented as two parts which are the global structure and corresponding deformations. Considering that distributions between the global structure and deformations of the 3D poses are different, we represented these two parts as the sparse and dense linear combination of 3D bases respectively. In addition, we proposed dual dictionary learning approach to learn two sets of 3D bases for representations of the 3D pose. Our dictionary learning model captures more implicit information as priors from limited training data. The effectiveness of our approach has been demonstrated on several evaluation benchmarks. In comparison, the proposed approach shown superior performance across all quantitative studies than all compared unsupervised learning approaches and even many competitive supervised methods. Especially, our approach shown more significant improvements in some categories with more deformations. For example, on the category ”Sit” of Human3.6M, our approach achieved state-of-the-art performance on estimation error metric with a wide margin (more than 66% and 28% improvements respectively comparing to all considered unsupervised and supervised learning approaches). In addition, the qualitative evaluation on in-the-wild images shown that our approach is able to estimate satisfactory 3D pose even without 3D annotation supervision. Future, we may extend our model to other non-rigid 3D object estimations, such as 3D face reconstruction, 3D hand pose estimation, etc.

APPENDIX A

The ALGORITHM TO SOLVE JOINT DICTIONARY LEARNING

The algorithm to solve Eq. (15) is presented here. For each training pose ${D_{m}}$, where $\forall m\in[1,N]$, we rewritten the cost function as

[TABLE]

where $\bm{B_{u}^{*}}$ and $\bm{B_{v}^{*}}$ are the concatenation of $\bm{B}_{uj}$ and $\bm{B}_{vj}$ respectively. The problems in (17) are solved by alternately updating strategy via projected gradient descent. The joint dictionary learning algorithm is summarized in Algorithm 2.

Acknowledgment

This research was supported by the National Nature Science Foundation of China (grant no 61671397).