Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited
Suryansh Kumar
TL;DR
This paper revisits a simple prior-free factorization method for Non-Rigid Structure from Motion, showing that with elementary modifications, its performance can be significantly improved to rival state-of-the-art algorithms.
Contribution
The authors identify hidden potential in a simple NRSfM algorithm and demonstrate how basic adjustments can enhance its accuracy and practical utility.
Findings
Performance improved by approximately 18% on benchmark datasets.
Elementary modifications can make simple methods competitive with complex algorithms.
Empirical results challenge prevailing views on the method's effectiveness.
Abstract
A simple prior free factorization algorithm \cite{dai2014simple} is quite often cited work in the field of Non-Rigid Structure from Motion (NRSfM). The benefit of this work lies in its simplicity of implementation, strong theoretical justification to the motion and structure estimation, and its invincible originality. Despite this, the prevailing view is, that it performs exceedingly inferior to other methods on several benchmark datasets \cite{jensen2018benchmark,akhter2009nonrigid}. However, our subtle investigation provides some empirical statistics which made us think against such views. The statistical results we obtained supersedes Dai {\it{et al.}}\cite{dai2014simple} originally reported results on the benchmark datasets by a significant margin under some elementary changes in their core algorithmic idea \cite{dai2014simple}. Now, these results not only exposes some unrevealed…
| Method | PTA[1] | CSF2[11] | PND[22] | BMM | Ours |
| Drink | 0.0287 | 0.0227 | 0.0037 | 0.0266 | 0.0119 (1.47%) |
| Pickup | 0.1939 | 0.1791 | 0.0372 | 0.1731 | 0.0198 (15.3%) |
| Yoga | 0.1243 | 0.1179 | 0.0140 | 0.1150 | 0.0129 (10.2%) |
| Stretch | 0.1035 | 0.1136 | 0.0156 | 0.1034 | 0.0144 (8.90%) |
| Dance | 0.2426 | 0.1877 | 0.1454 | 0.1864 | 0.1060 (8.04%) |
| Walking | 0.3761 | 0.1938 | 0.0465 | 0.1298 | 0.0882 (4.16%) |
| Face | 0.0489 | 0.0319 | 0.0165 | 0.0303 | 0.0179 (1.24%) |
| Shark | 0.2933 | 0.1117 | 0.0135 | 0.2311 | 0.0551 (17.6%) |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited
Suryansh Kumar
Computer Vision Lab, ETH Zürich, Switzerland
Abstract
A simple prior free factorization algorithm [9] is quite often cited work in the field of Non-Rigid Structure from Motion (NRSfM). The benefit of this work lies in its simplicity of implementation, strong theoretical justification to the motion and structure estimation, and its invincible originality. Despite this, the prevailing view is, that it performs exceedingly inferior to other methods on several benchmark datasets [14, 1]. However, our subtle investigation provides some empirical statistics which made us think against such views. The statistical results we obtained supersedes Dai et al.[9] originally reported results on the benchmark datasets by a significant margin under some elementary changes in their core algorithmic idea [9]. Now, these results not only exposes some unrevealed areas for research in NRSfM but also give rise to new mathematical challenges for NRSfM researchers. We argue that by properly utilizing the well-established assumptions about a non-rigidly deforming shape i.e, it deforms smoothly over frames [27] and it spans a low-rank space, the simple prior-free idea can provide results which is comparable to the best available algorithms. In this paper, we explore some of the hidden intricacies missed by Dai et. al. work [9] and how some elementary measures and modifications can enhance its performance, as high as approx. 18% on the benchmark dataset. The improved performance is justified and empirically verified by extensive experiments on several datasets. We believe our work has both practical and theoretical importance for the development of better NRSfM algorithms.
1 Introduction
Notation: The notation used in this paper is similar to Dai et al. work [9] unless otherwise stated.
Non-rigid Structure from Motion (NRSfM) is a well-known problem in geometric computer vision [5, 1, 9, 20, 18]. The goal of this problem is to reconstruct 3D structure of a deforming object using multiple frames. One of the most popular way to solve NRSfM is the matrix factorization approach. The matrix factorization approach to solve this problem dates back to 2000 [5] with no satisfactory solution in place until 2012. In the year 2012, Dai et al. [8] proposed a ground-breaking approach to solve NRSfM. This method for solving NRSfM is now considered as a classical work in NRSfM [9]. In that paper, the camera motion is estimated by imposing the null space constraint and the rank-3 positive semi-definite matrix cone constraint on the Gram matrix (\mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}). Further, nuclear norm minimization of the reshuffled shape matrix (\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}) was introduced to proffer stronger rank bound on the shape matrix for non-rigid shape estimation. The striking part of their work is that it not only challenged the myth of the inherent basis ambiguity in NRSfM [33] but also supplied a practical “prior-free" algorithm to solve NRSfM. Nonetheless, over years, it was observed that their remarkable theory performs poorly on benchmark datasets [22, 14]. In this paper, our goal is to make “prior-free idea” work well on real world scenarios.
Theoretically, the elementary idea of Dai et al. [9] conveniently encapsulates all the basic intuitions which are required to solve a general NRSfM problem. One may immediately argue on its performance when the deforming shape is composed of a union of low-rank subspace[19, 17, 36, 16, 15]. However, in this paper, we restrict our discussion to the classical representation of a NRSfM problem [5], without paying much attention to, how clustering benefits 3D reconstruction of the non-rigid object and other such notions of compact data representation. The reason for this choice is that the improvement in the performance of a classical baseline shall benefit the methods built on top of it.
The main purpose of this work is to uncover some of the unexplored mathematical intricacies in the prior free factorization approach to NRSfM, and improve on the idea supplied by Dai et al. [9]. Our exposition leads to the possible reasons for its inferior performance on the benchmark datasets [1, 14, 31]. It is shown in this paper that the rotation estimate using Dai et al. work [9] is not unique under the same model complexity prior (/rank), and they overlooked to utilize full correction matrix space [4]. Our investigation unveil the possibility of procuring motion that satisfies the well-known assumption of smooth non-rigid deformation of the object [27]. A simple search for the proper column-triplet (triads [4]) for the correction matrix (\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}) based on the smoothness of camera motion can indeed help improve the accuracy of the algorithm. Further, we argue that the weighted nuclear norm minimization of the shape matrix (\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}) is a far better choice than its global trace norm minimization. Lastly, due to our extensive analysis, we are able to posit some unsolved issues in NRSfM under “prior-free” idea which needs attention for further progress in this field.
In this paper, it is not claimed that we achieve state-of-the-art results on the benchmark datasets using our new approach. However, we empirically show that we can get very close to the best performing approaches and the difference is not very great, without the employment of complex and involved mathematical optimization [19, 22]. In this paper, we also argue that the inferior performance of “prior-free" method may not be due the proposed theoretical idea but because they overlooked some of the mathematical construction in their own formulation, and missed on properly utilizing the well-known assumptions about non-rigidly moving object i.e., smooth deformation [27] and low-rank shape [9]. Hence, the conclusion, understanding, and use of simple “prior-free” algorithm to NRSfM is not complete and precise. Through this work, we try to amend and nullify the prevailing perception about the “prior-free" approach, and how it can be used to its maximum potential. We feel that our paper touches some critical points which are essential to establish a theoretical closure to some of the elementary problems within the factorization approach to NRSfM.
Contribution: Firstly, our work postulates some rectification to the usage of “Intersection Method" [9] to compute camera motion. With the suitable example, we establish that the generalization made on the rotation matrix estimation by Dai et al. work [9] is not convincing and therefore, the knowledge about the strength of “Intersection theorem” is not completely exploited. Secondly, we provide an analytic solution to estimate suitable rotation using Intersection theorem and conjecture some challenges associated with it. Lastly, we propose a weighted nuclear norm minimization problem to estimate non-rigid 3D shape. Our approach shows a substantial improvement in the 3D reconstruction accuracy (nearly 18%). Moreover, we observed performance improvement in the case of noisy and missing trajectories §4.2 (under minor adjustment) using our method.
In this work, our attempt is to make the baseline method111By baseline, we mean the methods that solve NRSfM using its classical representation \mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}=\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}} that have withstood the test of time [30, 5]. more accurate, both in terms of understanding and performance, subject to the mathematical simplicity. To achieve this, we attempt to avoid the usage of complex mathematical notions such as union of independent subspace, dependent subspace representation [36, 19, 21], procrustean normal distribution [22], kernelization [10] etc. Hence, it is simple to understand the theoretical and practical justification of our method. We show that by applying simple but powerful logical and mathematical modifications to the prior free idea [9], we can get close to or even perform better at times than the best available algorithms on the benchmark datasets.
2 Representation and Motion Estimation
1. Classical Representation:
Tomasi and Kanade factorization method to structure-from-motion under orthographic camera projection appropriately summarizes the behavior of the 3D points over frames [30]. The relation between 3D shape, motion and its projection over frames was defined as
[TABLE]
where, \mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}\in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} is the measurement matrix formed by stacking all the image coordinates (\mathbf{x}=[\mathchoice{\mbox{\bf\displaystyle\tt u}}{\mbox{\bf\textstyle\tt u}}{\mbox{\bf\scriptstyle\tt u}}{\mbox{\bf\scriptscriptstyle\tt u}},\mathchoice{\mbox{\bf\displaystyle\tt v}}{\mbox{\bf\textstyle\tt v}}{\mbox{\bf\scriptstyle\tt v}}{\mbox{\bf\scriptscriptstyle\tt v}}]^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}) for ‘’ points along ‘’ rows i.e., total number of frames. \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}=\textrm{{blockdiagonal}}(\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{1},\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{2},..,\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}})\in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}} denotes the orthographic camera rotation matrix with each \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\in\mathbb{R}^{2\times 3} as per frame rotation. \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} represent the shape matrix with each row triplet as a 3D shape. This representation was later extended by Bregler et al. [5] to recover non-rigid 3D shapes. More concretely,
[TABLE]
The matrix ‘’ and ‘’ are composed of shape bases and shape coefficients respectively, with ‘’ as the number of shape bases. ‘’ denotes the kronecker product and ‘\mathchoice{\mbox{\bf\displaystyle\tt I}}{\mbox{\bf\textstyle\tt I}}{\mbox{\bf\scriptstyle\tt I}}{\mbox{\bf\scriptscriptstyle\tt I}}_{3}’ is a identity matrix. It is evident from the above formulation that the rank of \mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}\leq 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}} and also \textrm{rank}(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}})\leq 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}. However, is not a general rank 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}} matrix but own a special structure due to \mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}\otimes\mathchoice{\mbox{\bf\displaystyle\tt I}}{\mbox{\bf\textstyle\tt I}}{\mbox{\bf\scriptstyle\tt I}}{\mbox{\bf\scriptscriptstyle\tt I}}_{3} factor [9].
2. Null Space Representation of the Orthonormality Constraint:
An initial step in the factorization approach to NRSfM is to perform a rank 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}} decomposition of the measurement matrix via singular value decomposition (svd) i.e. \mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}=\hat{\Pi}\hat{\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}}. This is then followed by the estimation of Euclidean corrective matrix ‘’ to solve rotation and 3D structure. The main reason for such a procedure is due to the fact that the singular value decomposition of ‘’ matrix is not unique as any non-singular matrix \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} in between the two matrices and \hat{\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}} can form a valid factorization. Mathematically,
[TABLE]
Now, once we are able to solve correctly, then rotation and shape can be estimated using the above relations [5]. To solve , orthonormality constraints are imposed i.e. \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}=\mathchoice{\mbox{\bf\displaystyle\tt I}}{\mbox{\bf\textstyle\tt I}}{\mbox{\bf\scriptstyle\tt I}}{\mbox{\bf\scriptscriptstyle\tt I}}_{2}. Representing the \mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}^{\textrm{th}} double row of as \hat{\Pi}_{2\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}-1:2\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\in\mathbb{R}^{2\times 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} and \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3} as the \mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}^{\textrm{th}} column triplet of , then using Eq:(2) and Eq:(3) we can write
[TABLE]
Multiplying both sides by \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}} from right side gives
[TABLE]
where, \mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}=\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}. Using the algebraic relation \textrm{vec}(\mathchoice{\mbox{\bf\displaystyle\tt A}}{\mbox{\bf\textstyle\tt A}}{\mbox{\bf\scriptstyle\tt A}}{\mbox{\bf\scriptscriptstyle\tt A}}\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}) = (\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}\otimes\mathchoice{\mbox{\bf\displaystyle\tt A}}{\mbox{\bf\textstyle\tt A}}{\mbox{\bf\scriptstyle\tt A}}{\mbox{\bf\scriptscriptstyle\tt A}})\textrm{vec}(\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}), Dai et al. transformed these constraints (Eq:5) to a null space representation as follows:
[TABLE]
Using the above form and previous work in NRSfM [33], Dai et al. proposed the intersection theorem and supplied a SDP solution to estimate the \mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} matrix and the Euclidean corrective matrix \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} using svd().
Theorem 1
Intersection Theorem: Under non-generate and noise-free conditions, any correct solution of \mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} must lie in the intersection of the (2\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}^{2}-\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}) dimensional null-space of and a rank 3 positive semi-definite matrix cone i.e. \mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} must belong to
[TABLE]
Dai et al. solution to rotation: They proposed that once the \mathchoice{\mbox{\bf\displaystyle\tt Q}}{\mbox{\bf\textstyle\tt Q}}{\mbox{\bf\scriptstyle\tt Q}}{\mbox{\bf\scriptscriptstyle\tt Q}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} is solved, rather than solving for full Euclidean corrective matrix \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}, use svd() to extract rank 3 \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}. The solved \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3} can then be use to find (Eq:4) up to sign (\mathchoice{\mbox{\bf\displaystyle\tt c}}{\mbox{\bf\textstyle\tt c}}{\mbox{\bf\scriptstyle\tt c}}{\mbox{\bf\scriptscriptstyle\tt c}}_{\textrm{ik}}). The method quote “we adopt a simpler approach that directly computes the camera motion from single column-triplet \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} without need to fill in a big and full matrix”. Naturally, this single column-triplet is chosen to be the first column-triplet (\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{1}) of the matrix (see Fig:2(a)). Now, such strategy give rise to few legitimate concerns
- (a)
When each column triplet \{\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} qualifies for a suitable correction matrix, then why \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{1} has a high preference? Are we loosing useful information by such unwarranted preference? 2. (b)
Will each \{\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} provide the same solution to the rotation matrix? 3. (c)
Generally, most real world deformations are smooth in nature [27]. Whether such solution to rotation is good enough for the smooth deformation assumption?
Dai et al. overlooked all these intrinsic issues to solve rotation using their proposed intersection theorem.
Plausible Rectification: Our experiments show that Dai et al. [9] solution to rotation estimation actually aborted the useful information present in the \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}. Each of the ‘’ column triplets in (i.e. \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}) gives a possible rotation matrix which is different from each other (see Fig:(3)). Our empirical evaluations on several datasets show that the first column triplet is not always the best choice to estimate rotation. Hence, the details provided by Dai et al. work [9] is incomplete and there is a loss of generality with such procedure to estimate rotation under the well-known assumption of smooth deformation [27]. Fig:(4(a)) and Fig:(4(b)) provides few statistical results with comparison for both rotation and shape error estimate respectively. For clarity, we also provide the column triplet index that gives the better results for the corresponding data sequence and therefore, provides few counter-examples to such generalization.
Theoretically and practically, this result is of significant importance as it helps in inferring that the solution provided by “Intersection Theorem" has a lot of useful information left to be exploited completely and Dai et al. work ignored this. Also, it gives rise to some challenges that finding the best column triplet for \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} is not an easy task. With these results, we conjecture few problems for further research in NRSfM that are: (a) Can we find a best possible column triplet for the corrective matrix with a given rank prior ‘(\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}})’, or (b) At least can we put an upper bound on the value \mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}\subset\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}} such that there exists no such ‘’ for \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} which will provide better rotation and structure estimate. The problem seems hard keeping in view that the prior rank (\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}) in NRSfM factorization methods is an assumed approximation and it changes for different datasets to achieve better results.
A solution: In 2005, Brand. M [4] argued to use full correction matrix to estimate motion which in a way utilizes all the multiple estimates of column-triads of . Recently, Lee et al. [23] briefly mentions on the problems with motion estimates using [9]. In contrast, we use an analytical observation based on the smoothness and regularity222The term <<regularity>> is used in a loose sense (Mathematically). of the camera motion trajectory to filter \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times 3} to infer better ‘’. Let be a function that takes \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} as input and gives ‘’ as output using Intersection theorem. We estimate different \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}\in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}} for all the column triplets \{\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}\}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}, then compute smoothness of the camera motion for each \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}} as:
[TABLE]
By examining the smoothness of the camera motion for each \mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}_{\mathchoice{\mbox{\bf\displaystyle\tt k}}{\mbox{\bf\textstyle\tt k}}{\mbox{\bf\scriptstyle\tt k}}{\mbox{\bf\scriptscriptstyle\tt k}}}, we select the suitable rotation matrix for structure estimation (see Fig. 3). Our strategy to select smooth camera motion over frames based on Eq:(8) consistently supplied us with better performance than the previously proposed approach. We acknowledge that this is not a profound way to infer the best rotation, however, it does provide a possibility to deduce better rotation using “prior-free" approach which respects the well-known assumption of smooth deformation in NRSfM. Further, it helps endorse our claim on the generalization of rotation estimate by [9]. You may use the variable ‘\delta_{\mathchoice{\mbox{\bf\displaystyle\tt f}}{\mbox{\bf\textstyle\tt f}}{\mbox{\bf\scriptstyle\tt f}}{\mbox{\bf\scriptscriptstyle\tt f}}}’ Eq:(8) as a smoothness term in the final optimization (Eq:(11)) to further improve rotation, however, to show the competence within the “prior-free” idea [9], we stick to the classical two staged approach.
3 Structure Estimation
Once the rotation is estimated based on the smoothness of the camera motion [27], the next step is to solve for 3D structure. The block matrix method (BMM) by Dai et al. [9] proposed the following optimization problem to estimate the non-rigid low-rank shape.
[TABLE]
where, \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt 3}}{\mbox{\bf\textstyle\tt 3}}{\mbox{\bf\scriptstyle\tt 3}}{\mbox{\bf\scriptscriptstyle\tt 3}}\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} is a rearranged shape matrix with each row corresponds to the shape for that frame. The trace norm minimization on ‘\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}’ is enforced instead of ‘’ to provide a stronger rank bound on the shape matrix [9]. The second term in Eq:(9) enforces the re-projection error constraint. The function maps \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} to \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}}.
Dai et al. solution to shape: Following the work of Ma et al. [26] on rank minimization problems, Dai et al. [9] proposed a solution to the optimization in Eq:(9). The method enforces low-rank constraint on ‘\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}’ matrix and provide the solution by solving Eq:(9) via ADMM[3] using matrix shrinkage operator \mathcal{S}_{\lambda}(\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}) = \mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}\text{diag}(s_{\lambda}(\sigma))\mathchoice{\mbox{\bf\displaystyle\tt V}}{\mbox{\bf\textstyle\tt V}}{\mbox{\bf\scriptstyle\tt V}}{\mbox{\bf\scriptscriptstyle\tt V}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}, where = with = \big{\{}\sigma_{i}-\lambda~{}\text{if}~{}\sigma_{i}-\lambda>0 and 0 otherwise }.
Plausible Rectification: Despite the trace norm minimization provides a satisfactory solution to non-rigid structure estimation, it has some serious issues. The proposed solution to nuclear norm minimization problem (Eq:(9)) gives equal priority to each singular values, as a result, the shrinkage operator penalizes each singular value with the same quantity (). To estimate 3D structure of a non-rigidly deforming object using matrix factorization approach, we use a prior assumption that the shape lies in a low-rank subspace. Therefore, it’s not a better choice to penalize the major component of the shape data and its very minor component equally. Consequently, nuclear norm minimization of the shape matrix struggles to appropriately conserve the useful component of the non-rigidly deforming shape.
Truncated nuclear norm regularization can be a choice to handle such issues, however, it depends on the binary decision, hence not versatile in nature [35]. To really cater the behavior of the deformations based on its low-rank nature, we propose to use weighted nuclear norm minimization approach to solve for non-rigid structure [28, 12]. In contrast to the previous notation to the nuclear norm of the shape matrix i.e. \|\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}\|_{*}, we introduce a different notation for its weighted nuclear norm
[TABLE]
where \sigma_{\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}(.) denotes the \mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}^{\textrm{th}} singular value of \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}. We assume that the weights \Theta_{\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}’s are non-negative scalar i.e. \Theta_{\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}\geq 0 . Using this representation, we redefine the optimization proposed in the Eq:(9) as follows:
[TABLE]
The motivation for such formulation is quite clear, however, the proposed optimization (Eq:11) is generally non-convex, and is more difficult to solve than the nuclear norm minimization. Fortunately, recent results [34, 25, 12] in compressed sensing have shown that we can achieve an effective optimal solution to Eq:(11) in the case when 0\leq\Theta_{1}\leq\Theta_{2}\leq....\leq\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} §3.1.
3.1 Optimization
This section provides the mathematical derivation to the optimization proposed in Eq:(11). Our solution use the following theorems and proofs as stated and used in [34, 12, 7].
Theorem 2
For all \mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt m}}{\mbox{\bf\textstyle\tt m}}{\mbox{\bf\scriptstyle\tt m}}{\mbox{\bf\scriptscriptstyle\tt m}}\times\mathchoice{\mbox{\bf\displaystyle\tt n}}{\mbox{\bf\textstyle\tt n}}{\mbox{\bf\scriptstyle\tt n}}{\mbox{\bf\scriptscriptstyle\tt n}}}, denoted by \mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}=\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}\Sigma\mathchoice{\mbox{\bf\displaystyle\tt V}}{\mbox{\bf\textstyle\tt V}}{\mbox{\bf\scriptstyle\tt V}}{\mbox{\bf\scriptscriptstyle\tt V}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}, the SVD of it. The solution to \textrm{minimize}_{\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}}\|\mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}-\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}\|_{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}}^{2}+\|\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}\|_{\Theta,*}, with non-negative weight vector , its solution \hat{\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}} can be written as \hat{\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}}=\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}\hat{\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}}\mathchoice{\mbox{\bf\displaystyle\tt V}}{\mbox{\bf\textstyle\tt V}}{\mbox{\bf\scriptstyle\tt V}}{\mbox{\bf\scriptscriptstyle\tt V}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}, where \hat{\mathchoice{\mbox{\bf\displaystyle\tt B}}{\mbox{\bf\textstyle\tt B}}{\mbox{\bf\scriptstyle\tt B}}{\mbox{\bf\scriptscriptstyle\tt B}}} is the solution to the following optimization problem
[TABLE]
Theorem 3
If the singular values \sigma_{1}\geq....\geq\sigma_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} and the weights satisfy 0\leq\Theta_{1}\leq\Theta_{2}\leq....\leq\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} then the weighted nuclear norm minimization problem \textrm{minimize}_{\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}}\|\mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}-\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}\|_{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}}^{2}+\|\mathchoice{\mbox{\bf\displaystyle\tt X}}{\mbox{\bf\textstyle\tt X}}{\mbox{\bf\scriptstyle\tt X}}{\mbox{\bf\scriptscriptstyle\tt X}}\|_{\Theta,*} has a globally optimal solution
[TABLE]
where \mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}=\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}\Sigma\mathchoice{\mbox{\bf\displaystyle\tt V}}{\mbox{\bf\textstyle\tt V}}{\mbox{\bf\scriptstyle\tt V}}{\mbox{\bf\scriptscriptstyle\tt V}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}} is the SVD of , and is the generalized soft-thresholding operator with weight vector
[TABLE]
The readers are encouraged to refer to [34, 12] work for detailed derivations to the lemma’s leading to the proof of the theorems. In conclusion, if the weights satisfies non-descending order, not necessarily with the same value, the weighted nuclear norm minimization problem is still convex and optimal solution can be obtained using a soft-thresholding operator with different weights [34, 12].
3.2 Solution
We propose our solution to the optimization problem defined in Eq:(11) using alternating direction method of multipliers [3] (ADMM), a simple, fast but powerful algorithm used to solve many non-convex problems in computer vision and mathematical optimization. The ADMM algorithm decompose the original problem into several sub-problems, where each of them is solved separately by introducing Lagrange multipliers and penalty parameters to estimate convergence. Using the method of multipliers, the Augmented Lagrangian form for Eq:(11) is written as follows:
[TABLE]
here \mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} is a Lagrange multiplier and is the penalty parameter. The solution to each variable is obtained by solving the following subproblems over iterations (indexed with the variable ):
[TABLE]
[TABLE]
The Lagrange multiplier and the penalty parameter are updated as follows:
[TABLE]
refers to the maximum value of ‘’ and is an empirical constant (). The mathematical derivations to each sub-problems are provided in the supplementary material for reference. The closed form solution to the Eq:(17) is obtained by taking the derivative of Eq:(15) w.r.t variable ‘’ and equating it to zero i.e.,
[TABLE]
Similarly, rewriting the Eq:(15) treating \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp} as variable.
[TABLE]
In contrast to the previous form, the solution to Eq:(20) is not straight forward. To obtain a closed form solution to this problem, lets define a soft-thresholding function \mathcal{S}_{\tau}(\mathchoice{\mbox{\bf\displaystyle\tt\sigma}}{\mbox{\bf\textstyle\tt\sigma}}{\mbox{\bf\scriptstyle\tt\sigma}}{\mbox{\bf\scriptscriptstyle\tt\sigma}})=\textrm{sign}(\sigma).\textrm{max}(|\sigma|-\tau,0). Also, let [\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}},\Sigma,\mathchoice{\mbox{\bf\displaystyle\tt V}}{\mbox{\bf\textstyle\tt V}}{\mbox{\bf\scriptstyle\tt V}}{\mbox{\bf\scriptscriptstyle\tt V}}] be the singular value decomposition of (g(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}})-\frac{\mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}}{\rho}), then the optimal solution to Eq:(20) is given by:
[TABLE]
Here, is the weight assigned to the different singular values in the non-descending order based on its significance to the deformation data. For detail discussion on the initialization of weights kindly refer section §4.1.
4 Experiment and Discussion
To endorse our claim, we performed extensive experiments on both real and synthetic benchmark datasets [1, 14, 31]. We compared the performance of our algorithm against different state-of-the-art methods on these datasets [11, 22, 19]. Additionally, we unveil the substantial percentage boost in the reconstruction accuracy as high as 18% in comparison to the previous results reported for “simple prior-free" approach. For real-world applications to NRSfM, noisy data and missing feature tracks over frames are crucial, therefore, we also performed experiments to tackle such issues. To make the comparisons on noisy and missing data sequence, the experimental settings we used are same and consistent with Lee et al. work [22]. Experimental results on dense datasets and more rigorous cases of missing trajectories are provided in the supplementary material. Before we provide details on our performance analysis, lets discuss the variable initialization.
4.1 Initialization
Our algorithm has few parameters and variables to initialize. For all our experiments on different datasets, we initialize , , , , \mathchoice{\mbox{\bf\displaystyle\tt Y}}{\mbox{\bf\textstyle\tt Y}}{\mbox{\bf\scriptstyle\tt Y}}{\mbox{\bf\scriptscriptstyle\tt Y}}=\textrm{zeros}(\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}},3\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}) and the ‘’ values are kept same as Dai et al. method [9]. Practically, we considered the convergence of our optimization, if the gap {\text{max}}\|(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}-g(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}))\|_{\infty}<1e^{-8} or over iteration.
1. Structure initialization: Using the result of Liu et al. [24] on the uniqueness of minimizer for the rank minimization problem, we initialize the the 3D shape ‘’ as ‘’ = {\text{pinv}}(\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}})\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}} and \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}=g(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}). The pseudo-inverse solution to shape matrix provides a good enough initialization to our algorithm. Reader may refer to Dai et al. [9] and Valmadre et al. [32] work for detailed discussion on pseudo inverse solution to ‘’ in NRSfM.
2. Weight () initialization: It is well-known in NRSfM under factorization approach that the shape matrix lies in a low-rank space. Generally, the largest singular value of the shape matrix contains the most information about the non-rigid shape, therefore, while optimizing for the shape matrix, it’s illogical to treat each singular value equally. The singular values with major component must be penalized less and vice-versa. Using this inverse relation between singular values and its significance to the shape deformation modeling, we assign the weight () to be inversely propositional to the singular values of the shape matrix.
[TABLE]
where, is a positive number and , a very small positive number to avoid division by zero as some singular values are likely to be zero (low rank). We initialized the weights by substituting the pseudo-inverse initialization of ‘\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}’ i.e. using the relation \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp} = g(\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}) in the Eq:(22).
4.2 Performance Analysis
After a detailed discussion on the variable initialization and optimization, we present our experimental evaluation. We performed extensive experiments on both new and old benchmark datasets [1, 31, 14]. We report the quantitative result on the previous benchmark dataset using mean normalized 3D reconstruction error formulation i.e.
[TABLE]
where, \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}_{\textrm{ est}}, \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}_{\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}} are the estimated 3D shape and ground-truth 3D shape respectively. To keep our statistics consistent with the newly proposed NRSfM dataset, we used their error evaluation code to compute the robust root mean square error (RMSE) metric as proposed in Taylor et al. work [29]. For more details on NRSfM CVPR 2017 challenge dataset evaluation metric, kindly refer to Jensen et al. work [14].
1. Benchmark datasets: Most of the methods proposed in non-rigid structure from motion often use it to evaluate the performance of the algorithm. Loosely speaking, this dataset is composed of eight standard sequences namely Drink, Pickup, Yoga, Stretch, Dance, Walking, Face and Shark. The number of frames () to number of points () i.e. (\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}},\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}) set for these datasets are (1102, 41), (357, 41), (307, 41), (370, 41), (264, 75), (316, 40) and (240, 91) respectively. Table (1) show the statistical comparison of our approach in comparison to the other competing approaches for single body NRSfM. Our evaluation results clearly show a significant improvement in the reconstruction accuracy in comparison to the previously reported results for “prior-free” approach. Figure (5) show the qualitative reconstruction results w.r.t ground-truth on all of these sequences.
2. NRSfM challenge datasets: Jensen et al. recently released this dataset as a part of NRSfM competition held at CVPR 2017 [14]. This is a high quality challenging dataset divided into five categories based on the deformation type, namely, Articulated, Balloon, Paper, Stretch and Tearing. Each of these categories is again shot using six different camera paths namely circle, flyby, line, semi-circle, tricky and zig-zag. This dataset is significantly larger and diverse to really test the performance of a NRSfM algorithm’s. However, the dataset provides only a single frame ground-truth 3D for each of the five categories to test the algorithm. To estimate the reliability of our approach, we compared our performance against the best performing algorithm on this dataset. Table (2) show the quantitative results of our method. The performance clearly demonstrates the significant improvement in the accuracy using “prior-free” idea under our modification. It also help infer that without using complex mathematical notions, we can reach performance accuracy close to the state-of-the-art. Figure (6) show some qualitative results using our method.
3. Noisy data: The feature tracks captured from a real-world motion capture system is noisy most of the time. Therefore, to test the reliability and robustness of our new approach, we performed experiments by re-synthesizing the trajectories added with Gaussian noise. We introduced the Gaussian noise with standard deviation set as \sigma_{\textrm{noise}}=r*\textrm{max}\{|\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}|\}, where is varied from 0.05-0.25 [22]. Figure (7(a)) shows the variation in the normalized average 3D error for the stretch sequence using the performance of different algorithm recorded over 20 times. The plot clearly shows the robustness of our algorithm in comparison to other methods in the presence of large noise ratio’s.
4. Missing Data: In addition to the noisy data, the other problem with 3D reconstruction from a real video sequence is the missing trajectories over frames. We handle the missing trajectory quite robustly by incorporating a simple modification to the optimization proposed in Eq:(11). Let’s assume \tilde{\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}}\in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} is the incomplete measurement matrix and \mathchoice{\mbox{\bf\displaystyle\tt M}}{\mbox{\bf\textstyle\tt M}}{\mbox{\bf\scriptstyle\tt M}}{\mbox{\bf\scriptscriptstyle\tt M}}\in\{0,1\} is the mask matrix which indicates the presence or absence of the tracks over frames. Given \tilde{\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}}, , we first find a complete matrix using the following optimization
[TABLE]
The above optimization is a well studied optimization form. To keep things simple, we used Cabral et al. work [6] to estimate . The motive is to first solve for complete ‘’ to estimate camera motion using our rectified approach §2, and then solve for shape using the following cost function:
[TABLE]
Clearly, it’s just a minor adjustment to the proposed method based on the kind of data available in different situations. To evaluate our performance, we randomly set 30% of the data missing from the sequence same as Lee et al. work [22] for comparison. Figure (7(b)) shows the performance of our algorithm with missing data.
Discussion: Why not add the motion regularization in the final optimization and solve for both motion and shape? It’s definitely a valid argument. Nevertheless, we wanted to show the competence in a “prior free” way [9] which is to “solve for motion first using Intersection theorem and then solve for low-rank shape”, therefore, we avoided to add it in the final optimization. We showed that smooth solution [27] already exist within the solution to intersection theorem. Comprehensive analysis of our algorithm after adding motion regularization to solve the final optimization is left as an extension to the present idea.
5 Conclusion
With weighted nuclear norm minimization of the shape matrix and an analytic solution to the rotation matrix based on the smoothness of the camera motion [27], we witnessed that the prior-free idea performs almost as good as the best available algorithm’s. Without exploiting the “prior-free” idea [9] fully based on the well-known assumptions of smooth deformation of the non-rigid object and its low-rank shape, it may perform badly, which might be the reason that researchers have had poor results using it, even for the non-rigid objects that span a single linear subspace. Our work revealed the possibility of making “prior-free” [9] practically more accurate under the different conditions of measurement matrix with elementary modifications, and also conjecture some open problems. The accuracy of our algorithm on the benchmark datasets empirically validates that the “prior-free” theory is still a very powerful way to solve NRSfM and therefore, the proposition before the NRSfM researchers to consider is, it’s not the failure of the concept behind the prior-free idea for its inferior performance but, it’s possibly due to our inability to correctly cater, and cleverly exploit the arc of information and perspectives provided by it to solve NRSfM.
Acknowlegment: The author research work is supported by Google and ETH Zürich Foundation project number 2019-HE-323 (2).
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] I. Akhter, Y. Sheikh, S. Khan, and T. Kanade. Nonrigid structure from motion in trajectory space. In Advances in neural information processing systems , pages 41–48, 2009.
- 2[2] I. Akhter, Y. Sheikh, S. Khan, and T. Kanade. Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(7):1442–1456, 2011.
- 3[3] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning , 3(1):1–122, 2011.
- 4[4] M. Brand. A direct method for 3d factorization of nonrigid motion observed in 2d. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) , volume 2, pages 122–128. IEEE, 2005.
- 5[5] C. Bregler, A. Hertzmann, and H. Biermann. Recovering non-rigid 3d shape from image streams. In IEEE Conference on Computer Vision and Pattern Recognition , volume 2, pages 690–696. IEEE, 2000.
- 6[6] R. Cabral, F. De la Torre, J. P. Costeira, and A. Bernardino. Unifying nuclear norm and bilinear factorization approaches for low-rank matrix decomposition. In Proceedings of the IEEE International Conference on Computer Vision , pages 2488–2495, 2013.
- 7[7] J.-F. Cai, E. J. Candès, and Z. Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization , 20(4):1956–1982, 2010.
- 8[8] Y. Dai, H. Li, and M. He. A simple prior-free method for non-rigid structure-from-motion factorization. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , pages 2018–2025. IEEE, 2012.