A Kinematic Chain Space for Monocular Motion Capture

TL;DR

This paper introduces a novel monocular motion capture method using a kinematic chain space that does not rely on training data or specific camera/object motion, enabling reconstruction of diverse kinematic chains.

Contribution

The method projects observations into a kinematic chain space and employs nuclear norm optimization to enforce structural properties without requiring prior constraints or calibration.

Findings

Achieves state-of-the-art results on benchmark datasets.

Works with limited camera motion and unseen motions.

Applicable to various kinematic chains beyond human skeletons.

Abstract

This paper deals with motion capture of kinematic chains (e.g. human skeletons) from monocular image sequences taken by uncalibrated cameras. We present a method based on projecting an observation into a kinematic chain space (KCS). An optimization of the nuclear norm is proposed that implicitly enforces structural properties of the kinematic chain. Unlike other approaches our method does not require specific camera or object motion and is not relying on training data or previously determined constraints such as particular body lengths. The proposed algorithm is able to reconstruct scenes with limited camera motion and previously unseen motions. It is not only applicable to human skeletons but also to other kinematic chains for instance animals or industrial robots. We achieve state-of-the-art results on different benchmark data bases and real world scenes.

Tables

Table 1. Table 1: 3D error in m ​ m 𝑚 𝑚 mm for different sequences and data sets. The column walk35 shows the mean 3D error of all sequences containing walking motion from subject 35 in the CMU data base. jump refers to the jumping motion of subject 13/11 of the CMU data base. KTH means the football sequence of the KTH data set [ 12 ] . The column HE shows the 3D error for the HumanEva walking sequence [ 17 ] .

	walk35	jump	KTH	HE
[2]	228.68	210.14	108.91	106.92
[10]	264.75	186.70	114.03	102.99
Ours	18.94	36.50	53.10	44.36

Equations

$$\mathbf{X}=({\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_j).$$

$${\mathbf{X}}_k^{\prime }={\mathbf{P}}_k{\mathbf{X}}_k,$$

$$\mathbf{W}=\mathbf{P}\hat{\mathbf{X}},$$

$$\hat{\mathbf{W}}=\mathbf{W}-\mathbf{P}{\hat{\mathbf{X}}}_0.$$

$$\mathbf{W}=\mathbf{P}({\hat{\mathbf{X}}}_0+\mathbf{A}\mathbf{Q}).$$

$${\mathbf{b}}_k={\mathbf{p}}_r-{\mathbf{p}}_t=\mathbf{X}\mathbf{c},$$

$$\mathbf{c}=(0,\dots ,0,1,0,\dots ,0,-1,0,\dots ,0{)}^T,$$

$$\mathbf{B}=({\mathbf{b}}_1,{\mathbf{b}}_2,\dots ,{\mathbf{b}}_b).$$

$$\mathbf{B}=\mathbf{X}\mathbf{C},$$

$$\mathbf{X}=\mathbf{B}\mathbf{D}.$$

$$\mathbf{\Psi }={\mathbf{B}}^T\mathbf{B}=\left(\begin{array}{cccc}{l}_1^2& \cdot & \cdot & \cdot \\ \cdot & l_2^2& \cdot & \cdot \\ \cdot & \cdot & \ddots & \cdot \\ \cdot & \cdot & \cdot & l_b^2\end{array}\right).$$

$$trace(\mathbf{\Psi })=\sum _{i=1}^b{l}_i^2.$$

$$\Vert \mathbf{B}{\Vert }_{\ast }=trace(\sqrt{\mathbf{\Psi }})=const.$$

$$\mathbf{W}\mathbf{C}=\mathbf{P}({\hat{\mathbf{X}}}_0\mathbf{C}+\mathbf{A}\mathbf{Q}\mathbf{C})$$

$$\Vert \mathbf{B}{\Vert }_{\ast }=\Vert {\mathbf{B}}_1+{\mathbf{B}}_0{\Vert }_{\ast }=c_1,$$

$$\Vert {\mathbf{B}}_1{\Vert }_{\ast }+\Vert {\mathbf{B}}_0{\Vert }_{\ast }\ge \Vert {\mathbf{B}}_1+{\mathbf{B}}_0{\Vert }_{\ast }=c_1.$$

$$\Vert {\mathbf{B}}_1{\Vert }_{\ast }\ge c_1-\Vert {\mathbf{B}}_0{\Vert }_{\ast }=c,$$

$${\mathbf{B}}_1=\mathbf{AS}.$$

$$\Vert {\mathbf{B}}_1{\Vert }_{\ast }=\Vert \mathbf{A}{\Vert }_{\ast }.$$

$$\Vert \mathbf{A}{\Vert }_{\ast }\ge c.$$

$$\underset{\hat{\mathbf{A}}}{\min }$$

$$\Vert \mathbf{W}\mathbf{C}-\mathbf{P}(\hat{\mathbf{A}}\mathbf{S}+{\hat{\mathbf{B}}}_0){\Vert }_F=0.$$

$$\underset{{\mathbf{P}}_i}{\min }\Vert {\mathbf{X}}_i^{\prime }\mathbf{C}-{\mathbf{P}}_i(\mathbf{AS}+{\mathbf{B}}_0){\Vert }_F.$$

$$\bm{P}_{i}=\left(\begin{array}[]{cccc}p_{11}&p_{12}&p_{13}\\ p_{21}&p_{22}&p_{23}\\ \end{array}\right)$$

$$p_{11}^2+p_{12}^2+p_{13}^2-(p_{21}^2+p_{22}^2+p_{23}^2)=0$$

$$p_{11}{p}_{21}+p_{12}{p}_{22}+p_{13}{p}_{23}=0.$$

$$e=\frac{1}{j}\sum _{i=1}^j\Vert {\mathbf{x}}_i-{\hat{\mathbf{x}}}_i\Vert ,$$

Full text

1 Introduction

Monocular human motion capture is an important and large part of recent research. Its applications range from surveillance, animation, robotics to medical research. While there exist a large number of commercial motion capture systems, monocular 3D reconstruction of human motion plays an important role where complex hardware arrangements are not feasible or too costly.

Recent approaches to the non-rigid structure from motion problem [2, 10, 7, 16] achieve good results for laboratory settings. They are designed to work with tracked 2D points from arbitrary 3D point clouds. To resolve the duality of camera and point motion they require sufficient camera motion in the observed sequence. On the other hand in many applications (e.g. human motion capture, animal tracking or robotics) properties of the tracked objects are known. Exploiting known structural properties for non-rigid structure from motion problems is rarely considered e.g. by using example based modeling as in [5] or human bone lengths constancy in [23]. Recently, linear subspace training approaches have been proposed [15, 24, 1, 26, 23] and appear to efficiently represent human motion, even for 3D reconstruction from single images. However, they require extensive training on known motions which restricts them to reconstructions of the same motion category. Training based approaches cannot cover individual subtleties in the motion sufficiently well.

This paper closes the gap between non-rigid structure from motion and subspace-based human modeling. Similar to other approaches which depend on the seminal work of Bregler et al. [3], we decompose an observation matrix in three matrices corresponding to camera motion, transformation and basis shapes. Unlike other works that find a transformation which enforces properties of the camera matrices, we develop an algorithm that optimizes the transformation with respect to structural properties of the observed object. This reduces the amount of camera motion necessary for a good reconstruction. We experimentally found that even sequences without camera motion can be reconstructed. Unlike other works in the field of human modeling we propose to first project the observations in a kinematic chain space (KCS) before optimizing a reprojection error with respect to our kinematic model. Fig. 1 shows the mapping between the KCS and the representation based on 2D or 3D feature points. It is done by multiplication with matrices which implicitly encode a kinematic chain (cf. Sec. 3.1). This representation enables us to derive a nuclear norm optimization problem which can be solved very efficiently. Imposing a low rank constraint on a Gram matrix has shown to improve 3D reconstructions [7]. However, the method of [7] is only based on rotational constraints for the camera matrices. So the necessary amount of camera motion is still large. Our optimization in the KCS allows for the construction of a geometric constraint based on the topology of the underlying kinematic chain. So the required amount of camera motion is much lower.

We evaluate our method on different standard data bases (CMU MoCap [6], KTH [12], HumanEva [17]) as well as on our own data bases qualitatively and quantitatively. The proposed algorithm achieves state-of-the-art results and can handle problems like motion transfers and unseen motion. Due to the noise robustness of our method we can apply a CNN-based joint labeling algorithm [14] for RGB images as input data which allows us to directly reconstruct human poses from unlabeled videos. Although this method is developed for human motion capture it is applicable to other kinematic chains such as animals or industrial robots as shown in the experiments in Sec. 4.3.

Summarizing, our contributions are:

• •

We propose a method for 3D reconstruction of kinematic chains from monocular image sequences.

• •

An objective function based on structural properties of kinematic chains is derived that not only imposes a low-rank assumption on the shape basis but also has a physical interpretation.

• •

We propose using a nuclear norm optimization in a kinematic chain space.

• •

In contrast to other works our method is not limited to previously learned motion patterns and does not use strong anthropometric constraints such a-priorly determined bone lengths.

2 Related Work

The idea of decomposing a set of 2D points tracked over a sequence into matrices whose entries are identified with the parameters of shape and motion was first proposed by Tomasi and Kanade [19]. A generalization of this algorithm to deforming shapes was proposed by Bregler et al. [3]. They assume that the observation matrix can be factorized in two matrices representing camera motion and multiple basis shapes. After an initial decomposition is found by applying an SVD to the observation matrix they compute a transformation matrix by enforcing camera constraints. Xiao et al. [25] showed that the basis shapes of [3] are ambiguous. They solved this ambiguity by employing basis constraints on them. As shown by Akther et al. [2] these basis constraints are still not sufficient to resolve the ambiguity. Therefore, they proposed to use an object independent trajectory basis. Torresani et al. [20, 21, 22] proposed to use different priors on the transformation matrix such as additional rank constraints and gaussian priors. Gotardo and Martinez [9] built on the idea of [2] by applying the DCT representation to enforce a smooth 3D shape trajectory. Parallel to this work they proposed a solution that uses the kernel trick to also model nonlinear deformations [8] which cannot be represented by a linear combination of basis shapes. Hamsici et al. [10] combine the approaches of [9] and [8] by also assuming a smooth shape trajectory and apply the kernel trick to learn a mapping between the 3D shape and the 2D input data. Dai et al. [7] do not assume any prior knowledge about the scene such as smoothly moving cameras or points. Instead, they minimize the trace norm of the transformation matrix to impose a sparsity constraint. Since all these methods shall work for arbitrary non-rigid 3D objects, none of them utilizes knowledge about the underlying kinematic structure. Rehan et al. [16] were the first to define a temporary rigidity of reconstructed structures by factorizing only a small number consecutive frames. Therefore, they are able to model rigid structures to a small degree but due to their sliding window assumption the method is even more restricted to scenes with sufficient camera motion.

Several works consider the special case of 3D reconstruction of human motion from monocular images. A common approach is to previously learn base poses of the same motion category. They are then linearly combined for the estimation of 3D poses. To avoid implausible poses, most authors utilize properties of human skeletons to constrain a reprojection error based optimization problem. However, anthropometric priors like the sum of squared bone lengths [15], known limb proportions [24], known skeleton parameters [5] or previously trained joint angle constraints [1] all suffer from the fact that parameters have to be fixed or constrained. Zhou et al. [26] propose a convex relaxation of the commonly used reprojection error formulation to avoid the alternating optimization of camera and object pose. While many approaches try to reconstruct human poses from a single image using anthropometric priors, such constraints have rarely been used for 3D reconstruction from image sequences. Wandt et al. [23] used a temporal bone length constraint, which does not use known skeleton parameters. Zhou et al. [27] combined a deep neural network that estimates 2D landmarks with 3D reconstruction of the human pose. The restriction to a trained subset of possible human motions is the major downside of these subspace-based approaches.

In this paper we combine NR-SfM and human pose modeling without requiring previously learned motions. By using a representation that implicitly models the kinematic chain of a human skeleton our algorithm is capable to reconstruct unknown motion from labeled image sequences.

3 Estimating Camera and Shape

The $i$-th joint of a kinematic chain is defined by a vector $\bm{x}_{i}\in\mathbb{R}^{3}$ containing the $x$,$y$,$z$-coordinates of the location of this joint. By concatenating $j$ joint vectors we build a matrix representing the pose $\bm{X}$ of the kinematic chain

[TABLE]

The pose $\bm{X}_{k}$ in frame $k$ can be projected into the image plane by

[TABLE]

where $\bm{P}_{k}$ is the projection matrix corresponding to a weak perspective camera. For a sequence of $f$ frames, the pose matrices are stacked such that $\bm{W}=(\bm{X}^{\prime}_{1},\bm{X}^{\prime}_{2},\dots,\bm{X}^{\prime}_{f})^{T}$ and $\hat{\bm{X}}=(\bm{X}_{1},\bm{X}_{2},\dots,\bm{X}_{f})^{T}$. This implies

[TABLE]

where $\bm{P}$ is a block diagonal matrix containing the camera matrices $\bm{P}_{1,\dots,f}$ for the corresponding frame. In contrast to methods based on [3] we do not center the data at the mean. Instead, after an initial camera estimation we subtract a mean pose $\bm{X}_{0}$ from the measurement matrix by

[TABLE]

$\hat{\bm{X}}_{0}$ is a matrix of stacked mean poses similar to the construction of $\bm{W}$. It is used for initializing the algorithm and will be deformed during the optimization. The matrix $\bm{X}_{0}$ does not need to be the mean pose of the sequence or conform to anthropometric measures of the observed object. In our experiments we use a mean pose from various motions of the CMU data set. Different motions are reconstructed from the same mean pose. Following the approach of Bregler et al. [3] we decompose $\hat{\bm{W}}$ by Singular Value Decomposition to obtain a rank-$3K$ pose basis $\bm{Q}\in\mathbb{R}^{3K\times j}$. While [3] and similar works are optimizing a transformation matrix with respect to orthogonality constraints for camera matrices, we optimize the transformation matrix with respect to constraints based on a physical interpretation of the underlying structure. With $\bm{A}$ as transformation matrix for the pose basis we may then write

[TABLE]

In the following sections we will present how poses can be projected into the kinematic chain space (Sec. 3.1) and how we derive an optimization problem from it (Sec. 3.2). Combined with the camera estimation (Sec. 3.3) an alternating algorithm is presented in Sec. 3.4.

3.1 Kinematic Chain Space

To define a bone $\bm{b}_{k}$, a vector between the $r$-th and $t$-th joint is computed by

[TABLE]

where

[TABLE]

with $1$ at position $r$ and $-1$ at position $t$. The vector $\bm{b}_{k}$ has the same direction and length as the corresponding bone. Similarly to Eq. (1), a matrix $\bm{B}\in\mathbb{R}^{3\times b}$ can be defined containing all $b$ bones

[TABLE]

The matrix $\bm{B}$ is calculated by

[TABLE]

where $\bm{C}\in\mathbb{R}^{j\times b}$ is built by concatenating multiple vectors $\bm{c}$. Analogously to $\bm{C}$, a matrix $\bm{D}\in\mathbb{R}^{b\times j}$ can be defined that maps $\bm{B}$ back to $\bm{X}$:

[TABLE]

$\bm{D}$ is constructed similar to $\bm{C}$. Each column adds vectors in $\bm{B}$ to reconstruct the corresponding point coordinates. Note, that $\bm{C}$ and $\bm{D}$ are a direct result of the underlying kinematic chain. Therefore, the matrices $\bm{C}$ and $\bm{D}$ perform the mapping from point representation into the kinematic chain space and vice versa.

3.2 Trace Norm Constraint

One of the main properties of human skeletons is the fact that bone lengths do not change over time.

Let

[TABLE]

be a matrix with the squared bone lengths on its diagonal. From $\bm{B}\in\mathbb{R}^{3\times b}$ follows $rank(\bm{B})=3$. Thus $\bm{\Psi}$ has rank $3$. Note, if $\bm{\Psi}$ is computed for every frame we can define a stronger constraint on $\bm{\Psi}$. As bone lengths do not change for the same person the diagonal of $\bm{\Psi}$ remains constant.

Proposition 3.1**.**

The nuclear norm of $\bm{B}$ is invariant for any bone configuration of the same person.

Proof.

The trace of $\bm{\Psi}$ equals the sum of squared bone lengths (Eq. (11))

[TABLE]

From the assumption that bone lengths of humans are invariant during a captured image sequence the trace of $\bm{\Psi}$ is constant. The same argument holds for $trace(\sqrt{\bm{\Psi}})$. Therefore, we have

[TABLE]

∎

Since this constancy constraint is non-convex we will relax it to derive an easy to solve optimization problem. Using Eq. (9) we project Eq. (5) into the KCS which gives

[TABLE]

The remaining unknown is the transformation matrix $\bm{A}$. For better readability we define $\bm{B}_{0}=\bm{X}_{0}\bm{C}$ and $\bm{S}=\bm{Q}\bm{C}$.

Proposition 3.2**.**

The nuclear norm of the transformation matrix $\bm{A}$ for each frame has to be greater than a value $c$, which is constant for each frame.

Proof.

Let $\bm{B}=\bm{B}_{1}+\bm{B}_{0}$ be a decomposition of $\bm{B}$ into the initial bone configuration $\bm{B}_{0}$ and a difference to the observed pose $\bm{B}_{1}$. It follows that

[TABLE]

where $c_{1}$ is a constant. The triangle inequality for matrix norms gives

[TABLE]

Since $\bm{B}_{0}$ is known, it follows

[TABLE]

where $c$ is constant. $\bm{B}_{1}$ can be represented in the shape basis $\bm{S}$ (cf. Sec. 3) by multiplying it with the transformation matrix $\bm{A}$

[TABLE]

Since the shape base matrix $\bm{S}$ is a unitary matrix the nuclear norm of $\bm{B}_{1}$ equals

[TABLE]

By Eq. (17) follows that

[TABLE]

∎

Proposition 3.2 also holds for a sequence of frames. Let $\bm{\hat{A}}$ be a matrix built by stacking $\bm{A}$ for each frame and $\bm{\hat{B}}_{0}$ be defined similarly, we relax Eq. (20) and obtain the final formulation for our optimization problem

[TABLE]

Eq. (21) does not only define a low rank assumption on the transformation matrix. Due to our derivation and bone representation we showed that the nuclear norm is reasonable because it has a concise physical interpretation. More intuitively the minimization of the nuclear norm will give solutions close to a mean bone configuration $\bm{B}_{0}$ in terms of rotation of the bones, yet does not fix the solution to a predefined bone length which allows us to reconstruct arbitrary skeletons.

Moreover, Eq. (21) is a well studied problem which can be efficiently solved by common optimization methods such as Singular Value Thresholding (SVT) [4].

3.3 Camera

The objective function in Eq. (21) can also be optimized for the camera matrix $\bm{P}$. Since $\bm{P}$ is a block diagonal matrix, Eq. (21) can be solved block-wise for each frame. With $\bm{X}^{\prime}_{i}$ and $\bm{P}_{i}$ corresponding to the observation and camera at frame $i$ the optimization problem can be written as

[TABLE]

Considering the entries in

[TABLE]

we can enforce a weak perspective camera by the constraints

[TABLE]

and

[TABLE]

3.4 Algorithm

In the previous sections we derived an optimization problem that can be solved for the camera matrix $\bm{P}$ and transformation matrix $\bm{A}$ respectively. As both are unknown we propose algorithm 1 which alternatingly solves for both matrices. Initialization is done by setting all entries in the transformation matrix $\bm{A}$ to zero. Additionally, an initial bone configuration $\bm{B}_{0}$ is required. It has to roughly model a human skeleton but does not need to be the exact mean of the sequence.

4 Experiments

For the evaluation of our algorithm different benchmark data sets (CMU MoCap [6], HumanEva [17], KTH [12]) were used. As measure for the quality of the 3D reconstructions we calculate the Mean Per Joint Position Error (MPJPE) [11] which is defined by

[TABLE]

where $\bm{x}_{i}$ and $\hat{\bm{x}}_{i}$ correspond to the ground truth and estimated positions of the $i$-th joint respectively. By rigidly aligning the 3D reconstruction to the ground truth we obtain the 3D positioning error (3DPE) as introduced by [18]. To compare sequences of different lengths the mean of the 3DPE over all frames is used. In the following it is referred to as 3D error.

Additional to this quantitative evaluation we perform reconstructions of different kinematic chains in Sec. 4.3 and on unlabeled image sequences in Sec. 4.4. All animated meshes in this section are created using SMPL [13].

4.1 Evaluation on Benchmark Databases

To qualitatively show the drawbacks of learning-based approaches we reconstructed a sequence of a limping person. We use the method of [23] trained on walking patterns to reconstruct the 3D scene. Although the motions are very similar, the algorithm of [23] is not able to reconstruct the subtle motions of the limping leg. Fig. 4 shows the knee angle of the respective leg. The learning-based method reconstructs a periodic walking motion and cannot recover the unknown asymmetric motion. The proposed algorithm is able to recover the motion in more detail.

Since we do not require training data, we compare with other unsupervised works. For each sequence we created 20 random camera paths with low camera motion and compared our 3D reconstruction results with other state-of-the-art methods [2, 10]. Table 1 shows the 3D error in $mm$ for different sequences and data sets. For the entry walk35 we calculated the mean overall 3D errors of all 23 walking sequences from subject 35 in the CMU data base. The column jump shows the 3D error of a single jumping sequence. KTH means the football sequence of the KTH data set [12]. All these sequences are captured with limited camera motion. Methods like [2] and [10] require more camera motion and completely fail in these scenarios. Some of our reconstructions are shown in Figs. 2 and 3 for a walking and running motion.

4.2 Convergence

We alternatingly optimize the camera matrices (Eq. (21)) and transformation matrix (Eq. (22)). Since convergence of the algorithm cannot be guaranteed we show it by experiment. Fig. 5 shows the convergence of the reprojection error in pixel for a sequence from the CMU MoCap data base. However, the reprojection error only shows the convergence of the proposed algorithm but cannot prove that the 3D reconstructions will improve every iteration. We additionally estimated the convergence of the 3D error in Fig. 5. In most cases our algorithm converges to a good minimum in less than $3$ iterations. Further iterations do not improve the visual quality and only deform the 3D reconstruction less than $1mm$. The 3D error remains constant during camera estimation which causes the steps in the error plot.

Fig. 6 shows the computation time over the number of frames for three different sequences. The computation time mostly depends on the number frames and less on the observed motion. We use unoptimized Matlab code on a desktop PC for all computations.

4.3 Other Kinematic chains

Although our method was developed for the reconstruction of human motion it generalizes to all kinematic chains that do not include translational joints. In this section we show reconstructions of other kinematic chains such as people holding objects, animals and industrial robots.

In situations where people hold objects with both hands the kinematic chain of the body can be extended by another rigid connection between the two hands. Fig. 7 shows the reconstruction of the sword fighting sequence of the CMU data set. By simply adding another column to the kinematic chain space matrix $\bm{C}$ (cf. Sec. 3.1) the distance between the two hands is enforced to remain constant. The distance does not need to be known.

Fig. 8 shows a robot used for precision milling and the reconstructed 3D model as overlay. The proposed method is able to correctly reconstruct the robots motion. In Fig. 9 we reconstructed a more complex motion of a horse during show jumping. We used a simplified model of the bone structure of a horse. Also in reality the shoulder joint is not completely rigid. Despite these limitations the algorithm achieves plausible results.

4.4 Image Sequences

The proposed method is designed to reconstruct a 3D object from labeled feature points. In the former sections this was mainly done by setting and tracking them semi-interactive. In this section we will show that our method is also able to use the noisy output of a human joint detector. We use the pre-trained CNN of Newell et al. [14] to estimate the joints in a tennis sequence. Fig. 10 shows the joints estimated by the algorithm of [14] and our 3D reconstruction.

5 Conclusion

We developed a method for the 3D reconstruction of kinematic chains from monocular image sequences. By projecting into the kinematic chain space a constraint is derived that is based on the assumption that bone lengths are constant. This results in the formulation of an easy to solve nuclear norm optimization problem. It allows for reconstruction of scenes with little camera motion where other non-rigid structure from motion methods fail. Our method does not rely on previous training or predefined body measures such as known limb lengths. The proposed algorithm generalizes to the reconstruction of other kinematic chains and achieves state-of-the-art results on benchmark data sets.