Jumping Manifolds: Geometry Aware Dense Non-Rigid Structure from Motion
Suryansh Kumar
TL;DR
This paper introduces a geometry-aware dense non-rigid structure from motion algorithm that improves 3D shape reconstruction by leveraging both high- and low-dimensional Grassmann manifolds to better model surface deformations.
Contribution
It proposes a novel approach that jointly exploits high- and low-dimensional Grassmann manifolds for more accurate and robust non-rigid 3D reconstruction.
Findings
Effective handling of noise in 3D reconstruction.
Reconstruction accuracy comparable or superior to existing methods.
Abstract
Given dense image feature correspondences of a non-rigidly moving object across multiple frames, this paper proposes an algorithm to estimate its 3D shape for each frame. To solve this problem accurately, the recent state-of-the-art algorithm reduces this task to set of local linear subspace reconstruction and clustering problem using Grassmann manifold representation \cite{kumar2018scalable}. Unfortunately, their method missed on some of the critical issues associated with the modeling of surface deformations, for e.g., the dependence of a local surface deformation on its neighbors. Furthermore, their representation to group high dimensional data points inevitably introduce the drawbacks of categorizing samples on the high-dimensional Grassmann manifold \cite{huang2015projection, harandi2014manifold}. Hence, to deal with such limitations with \cite{kumar2018scalable}, we propose an…
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24
Figure 25
Figure 26
Figure 27| Dataset Method | MP [42] | PTA [2] | CSF1 [25] | CSF2[26] | DV [23] | DS [17] | SMSR [4] | SDG[34] | Ours |
| Face Sequence 1 | 0.2572 | 0.1559 | 0.5325 | 0.4677 | 0.0531 | 0.0636 | 0.1893 | 0.0443 | 0.0404 |
| Face Sequence 2 | 0.0644 | 0.1503 | 0.9266 | 0.7909 | 0.0457 | 0.0569 | 0.2133 | 0.0381 | 0.0392 |
| Face Sequence 3 | 0.0682 | 0.1252 | 0.5274 | 0.5474 | 0.0346 | 0.0374 | 0.1345 | 0.0294 | 0.0280 |
| Face Sequence 4 | 0.0762 | 0.1348 | 0.5392 | 0.5292 | 0.0379 | 0.0428 | 0.0984 | 0.0309 | 0.0327 |
| Actor Sequence 1 | 0.5226 | 0.0418 | 0.3711 | 0.3708 | - | 0.0891 | 0.0352 | 0.0340 | 0.0274 |
| Actor Sequence 2 | 0.2737 | 0.0532 | 0.2275 | 0.2279 | - | 0.0822 | 0.0334 | 0.0342 | 0.0289 |
| Paper Sequence | 0.0827 | 0.0918 | 0.0842 | 0.0801 | - | 0.0612 | - | 0.0394 | 0.0338 |
| T-shirt Sequence | 0.0741 | 0.0712 | 0.0644 | 0.0628 | - | 0.0636 | - | 0.0362 | 0.0386 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Jumping Manifolds: Geometry Aware Dense Non-Rigid Structure from Motion
Suryansh Kumar
Australian National University, CECS, Canberra, ACT, 2601
Abstract
Given dense image feature correspondences of a non-rigidly moving object across multiple frames, this paper proposes an algorithm to estimate its 3D shape for each frame. To solve this problem accurately, the recent state-of-the-art algorithm reduces this task to set of local linear subspace reconstruction and clustering problem using Grassmann manifold representation [34]. Unfortunately, their method missed on some of the critical issues associated with the modeling of surface deformations, for e.g., the dependence of a local surface deformation on its neighbors. Furthermore, their representation to group high dimensional data points inevitably introduce the drawbacks of categorizing samples on the high-dimensional Grassmann manifold [32, 31]. Hence, to deal with such limitations with [34], we propose an algorithm that jointly exploits the benefit of high-dimensional Grassmann manifold to perform reconstruction, and its equivalent lower-dimensional representation to infer suitable clusters. To accomplish this, we project each Grassmannians onto a lower-dimensional Grassmann manifold which preserves and respects the deformation of the structure w.r.t its neighbors. These Grassmann points in the lower-dimension then act as a representative for the selection of high-dimensional Grassmann samples to perform each local reconstruction. In practice, our algorithm provides a geometrically efficient way to solve dense NRSfM by switching between manifolds based on its benefit and usage. Experimental results show that the proposed algorithm is very effective in handling noise with reconstruction accuracy as good as or better than the competing methods.
1 Introduction
Non-rigid Structure-from-Motion (NRSfM), a problem where the task is to recover the three-dimensional structure of a deforming object from a set of image feature correspondences across frames. Any solution to this problem depends on the proper modeling of structure and an efficient estimation of motion , where denotes some structure manifold and denotes special Euclidean group which is a differentiable manifold [20]. Though, after Bregler et al. factorization framework to NRSfM [48], motion estimations are mostly relaxed to rotation estimation . Even after such relaxation, the problem still remains unsolved for any arbitrary motion. The main difficulty in NRSfM comes from the fact that both the camera and the object are moving and, along with it the object themselves are deforming, hence, it becomes difficult to distinguish camera motion from object motion using only image data. Despite such difficulties, many efficient and reliable solutions based on the priors are proposed to solve NRSfM. A reliable solution to this problem is important as it covers a wide range of applications from medical industry to the entertainment industry and many more.
To solve NRSfM, the algorithms proposed in the past can broadly be divided into two major classes 1) sparse NRSfM and 2) dense NRSfM. This classification is based on the number of feature points that the algorithm can efficiently process to model the deformation of the object. Although many reliable solution to this problem exists for sparse NRSfM [18, 37, 35, 3, 47, 42, 40, 26, 29], very few work have been done towards solving the dense NRSfM reliably and efficiently [23, 17, 34, 4]. Also, the existing solutions to dense NRSfM are computationally expensive and are mostly constrained to analyze the global deformation of the non-rigid shape [23, 17]. The basis for this gradual progress in dense NRSfM is perhaps due to its dependence on per pixel reliable correspondences across frames, and the absence of a resilient structure modeling framework to capture the local non-linearities. One may argue on the efficient motion estimation, however, from image correspondences, we can only estimate relative motion and reliable algorithms with solid theory exists to perform this task well [18, 40]. Also, with the recent progress in deep learning algorithms, per pixel correspondences can be achieved with a remarkable accuracy [46], which leaves structure modeling as a potential gray area in dense NRSfM to focus.
Very recently, Kumar et al. [34] has exploited the Grassmann manifold to model non-rigid surfaces in dense NRSfM. The key insight in their work is; even though the overall complexity of the deforming shape is high, each local deformation may be less complex [13, 14, 15, 16]. Using this idea, they proposed a union of local linear subspace approach to solve dense NRSfM problem. Nevertheless, their work overlooked on some of the intrinsic issues associated with the modeling of non-rigidly deforming surface. Firstly, their method represents each local linear subspace independently via a high-dimensional Grassmannian representation. Now, such representation may help reconstruct complex 3D deformation but can lead to wrong clustering, and it’s very important in joint reconstruction and clustering framework to have suitable clustering of subspaces, else reconstruction may suffer. Secondly, their approach to represent local non-linear deformation completely ignored the neighboring surfaces, which may result in an inefficient representation of the Grassmannians in the trajectory space. Thirdly, the representation of Grassmannians in the shape space adopted by [34] results in irredeemable discontinuity of the trajectories (see Fig.(2)). Hence, temporal representation of the set of shapes using Grassmannians seems not an extremely beneficial choice for modeling dense NRSfM on Grassmannian manifold111Purpose behind NRSfM is not the same as activity/action recognition. See supplementary material for a detail discussion on this.. Lastly, although the dense NRSfM algorithm proposed in [34] works better and faster than the previous methods, it depends on several manual parameters which are inadmissible for practical applications.
This paper introduces an algorithm that overcomes the aforementioned limitations with Kumar et al. method [34]. The main point we are trying to make is that; reconstruction and grouping of subspace on the same high dimensional Grassmann manifold seem an unreasonable choice. Even recent research in the Riemannian geometry has shown that the low-dimensional representation of the corresponding high dimensional Grassmann manifold is more favorable for grouping Grassmannians [32, 31]. So, we formulate dense NRSfM in a way that it takes advantage of both high and low dimensional representation of Grassmannians i.e., perform reconstruction in the original high-dimension and cluster subspace in its lower-dimension representation.
We devise an unsupervised approach to efficiently represent the high-dimensional non-rigid surface on a lower dimensional Grassmann manifold. These low-dimensional Grassmannians are represented in such a way that it preserves the local structure of the surface deformation in accordance with its neighboring surfaces when projected. Now, these low-dimensional Grassmannians serves as a potential representative for its high-dimensional Grassmannians for suitable grouping, which subsequently help improve the reconstruction and representation of the Grassmannians on the high-dimensional Grassmann manifold, hence, the term Jumping Manifolds (MoJu). Further, we drop the temporal grouping of shapes using Grassmannians to discourage the discontinuity of trajectories (see Fig.(2)).
In essence, our work is inspired by [34] and is oriented towards settling its important limitations. Moreover, in contrast to [34], we capture the notion of dependent local subspace in a union of subspace algorithm [39] via Grassmannian modeling. The algorithm we proposed is an attempt to supply a more efficient, reliable and practical solution to this problem. Our formulation gives an efficient framework for modeling dense NRSfM on the Grassmann manifold than [34]. Experimental results show that our method is as accurate as other algorithms and is numerically more efficient in handling noise. The main contributions of our work are as follows:
- •
An efficient framework for modeling non-rigidly deforming surface that exploits the advantage of Grassmann manifold representation of different dimensions based on its geometry.
- •
A formulation that encapsulates the local non-linearity of the deforming surface w.r.t its neighbors to enable the proper inference and representation of local linear subspaces.
- •
An iterative solution to the proposed cost function based on ADMM [8], which is simple to implement and provide results as good as the best available methods. Additionally, it helps improve the 3D reconstruction substantially, in the case of noisy trajectories.
2 Related Work
A working solution to NRSfM was first introduced in the seminal work by Bregler et al. [9] which is an extension to Tomsai-Kannade rigid factorization method[48]. Although the problem still remains unsolved for arbitrary deformations, many profound works have been done to achieve a reliable solution to this problem under some or the other prior assumptions about the object [18, 3, 43, 40, 50, 37, 35, 23]. Since the literature on this topic is very extensive, we review the works that are of close relevance to the dense NRSfM methods under classical setting222By classical setting, we mean without using RGB-D or 3D template..
Earlier attempts to solve dense NRSfM used piecewise reconstruction of the shape parts which were further processed via a stitching step to get a global 3D shape [12, 45]. To our knowledge, Garg et al. variational approach [24] was the first to propose and demonstrate per pixel dense NRSfM algorithm without any 3D template prior. This method introduced a discrete total variational constraint with trace norm constraint on the global shape, which resulted in a biconvex optimization problem. Despite the algorithm outstanding results, it’s computationally very expensive and needs a GPU to provide the solution.
In contrast, Dai et al. extended his simple prior free approach to solve dense NRSfM problem [18, 17]. The algorithm proposed a spatial-temporal formulation to solve the problem. The author revisits the temporal smoothness term from [18] and integrate it with a spatial smoothness term using the Laplacian of the non-rigid shape. The resultant optimization leads to a series of least squares to be minimized which makes it extremely slow to process. Recently, Kumar et al. modeled this problem on the Grassmann manifold [34]. Their work extended the spatiotemporal multi-body framework to solve dense NRSfM [37]. The algorithm demonstrated that such an approach is more efficient, faster and accurate than all the other recent approaches to solve dense NRSfM task [24, 17, 4].
Consecutive frame-based approach has recently shown some promising results to solve dense 3D reconstruction of a general dynamic scene including non-rigid object [36, 44]. Nevertheless, motion segmentation, triangulation, as rigid as possible constraint and scale consistency quite often breaks down for the deforming object over frames. Therefore, dense NRSfM becomes extremely challenging for such algorithms. Not long ago, Gallardo et al. combined shading, motion and generic physical deformation to model dense NRSfM [21].
3 Preliminaries
In this paper, \|.\|_{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}}, denotes the Frobenius norm and nuclear norm respectively. represent the notion of norm on the Grassmann manifold. Single angle bracket denotes the Euclidean inner product. For ease of understanding and completeness, in this section, we briefly review few important definitions related to the Grassmann manifold. Firstly, a manifold is a topological space that is locally similar to the Euclidean space. Out of several manifolds, the Grassmann manifold is a topologically rich non-linear manifold, each point of which represent the set of all right invariant subspace of the Euclidean space [19, 1, 34].
Definition 1**.**
The Grassmann manifold, denoted by \mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}), consists of all the linear -dimensional subspace embedded in a ‘’ dimensional Euclidean space \mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}} such that 0\leq\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}\leq\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}} [Absil et al., 2009] [1].
A point ‘’ on the Grassmann manifold can be represented by \mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}} matrix whose columns are composed of orthonormal basis. The space of such matrices with orthonormal columns is a Riemannian manifold such that \Phi^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}\Phi=\mathbf{I}_{\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}}, where \mathbf{I}_{\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}} is a \mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}} identity matrix.
Definition 2**.**
Grassmann manifold can be embedded into the space of symmetric matrices via mapping \Pi:\mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}})\mapsto\textrm{Sym}(\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}),\Pi(\Phi)=\Phi\Phi^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}, where is a Grassmann point [28, 30]. Given two Grassmann points and , then the distance between them can be measured using the projection metric d_{g}^{2}(\Phi_{1},\Phi_{2})=0.5\|\Pi(\Phi_{1})-\Pi(\Phi_{2})\|_{\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}}^{2} [28].
In the past, these two properties of Grassmann manifold has been used in many computer vision applications [28, 10, 34]. Second definition is very important as it allows to measure the distance on the Grassmann manifold, hence, ) forms a metric space. We used these properties in the construction of our formulation. For comprehensive details on this topic readers may refer to [28].
4 Problem Formulation
Let ‘’ be the total number of feature points tracked across ‘’ frames. Concatenating these 2D coordinates of each feature points for all frames across the columns of a matrix gives ‘’ \in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} matrix. This matrix is popularly known as measurement matrix [48]. Our goal is, given the image measurement matrix, estimate the camera motion and 3D coordinates of every 2D feature points across all frames.
We start our formulation with the classical representation to NRSfM i.e. \mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}=\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}, where, \mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}}\in\mathbb{R}^{2\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times 3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}} is a block diagonal rotation matrix with each block as a orthographic rotation matrix, and \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} as the 3D structure matrix. With such a representation, the entire problem simplifies to the estimation of correct rotation matrix ‘’ and structure matrix ‘’ such that the above relation holds. Following the assumption of the previous work [34], we estimate the rotation using Intersection method [18]. As a result, the task reduces to composing of an efficient algorithm that correctly models the surface deformations and provide better reconstruction results. Recent algorithms in NRSfM have demonstrated that clustering benefits reconstruction and vice-versa, however, the existing framework to employ this idea is not scalable to millions of points. To establish this idea for dense NRSfM, Kumar et al. [34] used LRR on Grassmannian manifold. Using the similar notions, we model dense deforming surface using Grassmannian representation to provide more reliable and accurate solution.
In the following subsection, we first introduce the Grassmannian representation of the surface and how to project these Grassmannians onto the lower dimension Grassmann manifold by preserving the neighboring information. In the later subsection, we use these representations to formulate the overall cost function for solving dense NRSfM problem.
4.1 Grassmannian representation
Let ‘\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}’ \in\mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}) be a Grassmann point representing the \mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}^{\textrm{th}} local linear subspace spanned by \mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}^{\textrm{th}} set of columns of ‘’. Using this notion, we decompose the entire trajectories of the structure into a set of ‘’ Grassmannians \xi=\{\Phi_{1},\Phi_{2},\Phi_{3},....,\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\}. Now, such a representation treats each subspace independently and therefore, its low-dimensional linear representation may not be suitable to capture the surface dependent non-linearity. To properly represent Grassmannian which respects the neighboring non-linearity in low-dimension, we introduce a different strategy to model non-rigid surface in low-dimension. For now, let \Delta\in\mathbb{R}^{{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\times\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}} be a matrix that maps ‘\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}’ \in\mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}) to ‘\phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}’ \in\mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}) such that \tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}<{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}. Mathematically,
[TABLE]
Its quite easy to examine that is not a orthogonal matrix and, therefore, does not qualifies as a potential point on a Grassmann manifold. However, by performing a orthogonal-triangular (QR) decomposition of , we estimate the new representative of on the Grassmann manifold of ‘\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}’ dimension.
[TABLE]
Here, is a function that returns the QR decomposition of the matrix. The \Theta_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\in\mathbb{R}^{\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}} is an orthogonal matrix and \mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}} is the upper triangular matrix333Note: The value of \tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\geq\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}, Use [\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}},\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}]=\textbf{qr}(\phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}},0) in MATLAB to get a square \mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}} matrix (\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}}). Using Eq.(2), we represent the equivalence of \Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}} in low dimension as
[TABLE]
where, \Omega_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}=\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\mathchoice{\mbox{\bf\displaystyle\tt U}}{\mbox{\bf\textstyle\tt U}}{\mbox{\bf\scriptstyle\tt U}}{\mbox{\bf\scriptscriptstyle\tt U}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}^{-1}\in\mathbb{R}^{{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\times\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}}}. The key-point to note is that both \Theta_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}} and \phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}} has the same column space. In principle such a representation is useful however, it does not serve the purpose of preserving the non-linearity w.r.t its neighbors. In order to encapsulate the local dependencies (see Fig.(3), Fig.(4)), we further constrain our representation as:
[TABLE]
The parameter ‘\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}’ accommodate the similarity knowledge between the two Grassmannians. Using the Definition(2) and Eq.(3), we further simplify Eq.(4) as
[TABLE]
where, \Lambda_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}\in Sym({d}). The parameter ‘\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}’ (similarity graph) is set as exp(-d_{g}^{2}(\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}},\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}})) with as the projection metric (see Definition (2)). Eq.(5) is an unconstrained optimization problem and its solution may provide a trivial solution. To estimate the useful solution, we further constrain the problem. Using \mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}^{\mathchoice{\mbox{\bf\displaystyle\tt t}}{\mbox{\bf\textstyle\tt t}}{\mbox{\bf\scriptstyle\tt t}}{\mbox{\bf\scriptscriptstyle\tt t}}\mathchoice{\mbox{\bf\displaystyle\tt h}}{\mbox{\bf\textstyle\tt h}}{\mbox{\bf\scriptstyle\tt h}}{\mbox{\bf\scriptscriptstyle\tt h}}} Grassmann point ‘\Omega_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}’ and its neighbors, expand Eq.(5). By performing some simple algebraic manipulation, Eq.(5) reduces to
[TABLE]
where, \lambda_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}=\sum_{\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt j}}{\mbox{\bf\textstyle\tt j}}{\mbox{\bf\scriptstyle\tt j}}{\mbox{\bf\scriptscriptstyle\tt j}}}. Constraining the value of Eq.(6) to 1 provides the overall optimization for an efficient representation of the local non-rigid surface on the Grassmann manifold.
[TABLE]
Its easy to verify that the matrix and \big{(}\sum_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\lambda_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\Omega_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\Omega_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}\big{)} are symmetric and positive semi-definite, and therefore, the above optimization can be solved as a generalized eigen value problem —refer supplementary material for details.
4.2 Dense NRSfM formulation
To solve the dense non-rigid structure from motion with the representation formulated in the previous sub-section §4.1, we propose to jointly optimize the objective function over the 3D structure and its local group representation. In order to build the overall objective function, we introduce each constraint equation one by one for clear understanding of our overall cost function.
[TABLE]
The first term constrain the 3D structure such that it satisfies the re-projection error.
[TABLE]
The second term caters the global assumption about the non-rigid object; that is the overall shape matrix is low-rank. To establish this assumption, we perform rank minimization of the shape matrix. Although the rank minimization of a matrix is NP-hard, it’s relaxed to nuclear norm minimization to find an approximate solution. This term mainly penalizes the total number of independent shape required to represent the shape. The choice of minimizing \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}\times\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}} instead to \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} is inspired from Dai et al.’s work [18]. Since the dense deforming shape is composed of several local linear low-dimensional subspace, the global constraint (Eq.(9)) may not reflect their local dependency. Therefore, in order to introduce the local subspace constraint on the shape, we use the notion of self-expressiveness on the non-linear Grassmann manifold space.
[TABLE]
Here, we define f:\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}}\mapsto\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}^{\sharp}\in\mathbb{R}^{3\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}\times\mathchoice{\mbox{\bf\displaystyle\tt F}}{\mbox{\bf\textstyle\tt F}}{\mbox{\bf\scriptstyle\tt F}}{\mbox{\bf\scriptscriptstyle\tt F}}} and \mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} as the coefficient matrix. We know from the literature that the Grassmann manifold is isometrically equivalent to the symmetric idempotent matrix [11]. So, we embed the Grassmann manifold into symmetric matrix manifold to define the self-expressiveness. Let \tilde{\xi}=\{\Theta_{1},\Theta_{2},...,\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\} be the set of Grassmannians on a low-dimensional Grassmann manifold. The elements of are the projection of high-dimensional Grassmannian representation of the columns of ‘’ matrix. Let {\mathchoice{\raisebox{0.0pt}{\displaystyle\chi}}{\raisebox{0.0pt}{\textstyle\chi}}{\raisebox{0.0pt}{\scriptstyle\chi}}{\raisebox{0.0pt}{\scriptscriptstyle\chi}}}=\{(\Theta_{1}\Theta_{1}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}),(\Theta_{2}\Theta_{2}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}),...,(\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\Theta_{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}^{\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}})\} be its embedding onto symmetric matrix manifold. Using such embedding techniques we re-write Eq.(10) as
[TABLE]
where, \tilde{\mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}}\in\mathbb{R}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}\times\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} and {\mathchoice{\raisebox{0.0pt}{\displaystyle\chi}}{\raisebox{0.0pt}{\textstyle\chi}}{\raisebox{0.0pt}{\scriptstyle\chi}}{\raisebox{0.0pt}{\scriptscriptstyle\chi}}}\in\mathbb{R}^{\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\times\tilde{\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}}\times\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}} denotes the coefficient matrix of Grassmannians and structure tensor respectively, with as the total number of Grassmannians. Generally, \mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}<<\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}, which makes such representation scalable.
The third term we introduce is composed of few constraint functions that provides a way to group Grassmannians and recover 3D shape simultaneously. Let \in\mathbb{R}^{1\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} be an ordering vector that contains the index of columns of . Our function definition is of the form . Using it, we define the function , , and as follows:
[TABLE]
[TABLE]
[TABLE]
[TABLE]
Intuitively, the first function () uses the ordering vector \in\mathbb{R}^{1\times\mathchoice{\mbox{\bf\displaystyle\tt P}}{\mbox{\bf\textstyle\tt P}}{\mbox{\bf\scriptstyle\tt P}}{\mbox{\bf\scriptscriptstyle\tt P}}} to refine the grouping of the trajectories for suitable Grassmannian representation. The second function () projects the Grassmannians to a lower dimension in accordance with the neighbors using Eq.(7). The third function () uses the projected Grassmannians to assign proper labeling to the Grassmann points and update the given ordering vector using spectral clustering. The fourth function () uses the group of trajectories to reconstruct back the set of local surface. , {\xi_{\mathchoice{\mbox{\bf\displaystyle\tt v}}{\mbox{\bf\textstyle\tt v}}{\mbox{\bf\scriptstyle\tt v}}{\mbox{\bf\scriptscriptstyle\tt v}}}} are the singular values and right singular vector matrices in the high-dimension.
Objective Function: Combining all the above terms and constraints provides our overall cost function.
[TABLE]
where \mathbf{P}_{\mathchoice{\mbox{\bf\displaystyle\tt o}}{\mbox{\bf\textstyle\tt o}}{\mbox{\bf\scriptstyle\tt o}}{\mbox{\bf\scriptscriptstyle\tt o}}} vector contains the initial ordering of the columns of ‘’ and ‘’. The function provides the ordering index to rearrange the columns of ‘’ matrix to be consistent with ‘’ matrix. This is important because, grouping the set of columns of ‘’ over iteration, disturbs its initial arrangements.
5 Solution
The optimization proposed in Eq.(16) is a coupled optimization problem. Several methods of Bi-level optimization can be used to solve such minimization problem [6, 27]. Nevertheless, we propose ADMM [8] based solution due to its application in many non-convex optimization problems. The key point to note is that one of our constraint is composed of separate optimization problem i.e., the solution to Eq.(7), and therefore, we cannot directly embed the constraint to the main objective function. Instead, we only introduce two Lagrange multiplier to concatenate a couple of constraints back to the original objective function. The remaining constraints are enforced over iteration. To decouple the variable \tilde{\mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}} from , we introduce auxiliary variable \tilde{\mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}}=\mathchoice{\mbox{\bf\displaystyle\tt Z}}{\mbox{\bf\textstyle\tt Z}}{\mbox{\bf\scriptstyle\tt Z}}{\mbox{\bf\scriptscriptstyle\tt Z}}. We apply these operations to our optimization problem to get the following Augmented Lagrangian form:
[TABLE]
Note that \tilde{\mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}} provides the information about the subspace, not the vectorial points. However, we have the chart of the trajectories and its corresponding subspace. Once, we group the trajectories based on \tilde{\mathchoice{\mbox{\bf\displaystyle\tt C}}{\mbox{\bf\textstyle\tt C}}{\mbox{\bf\scriptstyle\tt C}}{\mbox{\bf\scriptscriptstyle\tt C}}}, provides new Grassmann sample corresponding to each group. The definition of and is provided in Eq.(7) and Eq.(14) respectively. More generally, the solution to the optimization in Eq.(7) is obtained by solving it as a generalized eigenvalue problem. To keep the order of columns of ‘’ matrix consistent with ‘’ matrix provides the ordering index. We provide the implementation details of our method with suitable MATLAB commands in the Algorithm Table (1). For details on the derivation to each sub-problem, kindly refer to the supplementary material.
6 Initialization and Evaluation
We performed experiments and evaluation on the available standard benchmark datasets [23, 49, 7]. To keep our evaluations consistent with the previous methods, we compute the mean normalized 3D reconstruction error of the estimated shape ‘\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}_{\textrm{est}}’ after convergence as
[TABLE]
here ‘\mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}_{\mathchoice{\mbox{\bf\displaystyle\tt G}}{\mbox{\bf\textstyle\tt G}}{\mbox{\bf\scriptstyle\tt G}}{\mbox{\bf\scriptscriptstyle\tt G}}\mathchoice{\mbox{\bf\displaystyle\tt T}}{\mbox{\bf\textstyle\tt T}}{\mbox{\bf\scriptstyle\tt T}}{\mbox{\bf\scriptscriptstyle\tt T}}}’ denotes the ground-truth 3D shape matrix.
Initialization:
We used Intersection method [18] to estimate the rotation matrix and initialize \mathchoice{\mbox{\bf\displaystyle\tt S}}{\mbox{\bf\textstyle\tt S}}{\mbox{\bf\scriptstyle\tt S}}{\mbox{\bf\scriptscriptstyle\tt S}}=\textbf{pinv}(\mathchoice{\mbox{\bf\displaystyle\tt R}}{\mbox{\bf\textstyle\tt R}}{\mbox{\bf\scriptstyle\tt R}}{\mbox{\bf\scriptscriptstyle\tt R}})\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}. The initial grouping of the trajectories or columns of is done using k-means++ algorithm [5]. These groups are then used to initialize \mathbf{P}_{\mathchoice{\mbox{\bf\displaystyle\tt o}}{\mbox{\bf\textstyle\tt o}}{\mbox{\bf\scriptstyle\tt o}}{\mbox{\bf\scriptscriptstyle\tt o}}}, and the Grassmann points \{\Phi_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}}\}_{\mathchoice{\mbox{\bf\displaystyle\tt i}}{\mbox{\bf\textstyle\tt i}}{\mbox{\bf\scriptstyle\tt i}}{\mbox{\bf\scriptscriptstyle\tt i}}=1}^{\mathchoice{\mbox{\bf\displaystyle\tt K}}{\mbox{\bf\textstyle\tt K}}{\mbox{\bf\scriptstyle\tt K}}{\mbox{\bf\scriptscriptstyle\tt K}}}\in\xi via subset of singular vectors. To represent the Grassmannians in the lower-dimension, we solve Eq.(7) to initialize and store corresponding singular values. The similarity matrix or graph in Eq.(7) is build using the distance measure between the Grassmannians in the embedding space §4.1.
1. Results on synthetic Face dataset: The synthetic face dataset is composed of four distinct sequence [23] with 28,880 feature points tracked over multiple frames. Each sequence captures the human facial expression with a different range of deformations and camera motion. Sequence 1 and Sequence 2 are 10 frame long video with rotation in the range and respectively. Sequence 3 and Sequence 4 are 99 frame long video that contains high frequencies and low frequencies rotation respectively which captures real human facial deformations. Table (1) shows the statistical results obtained on these sequences using our algorithm. For qualitative results on these sequences kindly refer to the supplementary material.
2. Results on Paper and T-shirt dataset: Varol et al. introduced ‘kinect_paper’ and ‘kinect_tshirt’ datasets to test the performance of NRSfM algorithm under real conditions [49]. This dataset provides sparse SIFT [41] feature tracks and noisy depth information captured from Microsoft Kinect for all the frames. As a result, to get dense 2D feature correspondences of the non-rigid object for all the frames becomes difficult. To circumvent this issue, we used Garg et al. [22] algorithm to estimate the measurement matrix. To keep the numerical comparison consistent with the previous work in dense NRSfM [34], we used the same coordinate range for tracking the features. Numerically, its \mathchoice{\mbox{\bf\displaystyle\tt x}}{\mbox{\bf\textstyle\tt x}}{\mbox{\bf\scriptstyle\tt x}}{\mbox{\bf\scriptscriptstyle\tt x}}_{\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}} = , \mathchoice{\mbox{\bf\displaystyle\tt y}}{\mbox{\bf\textstyle\tt y}}{\mbox{\bf\scriptstyle\tt y}}{\mbox{\bf\scriptscriptstyle\tt y}}_{\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}} = rectangular window across 193 frames for kinect_paper sequence. For kinect_tshirt sequence, we considered rectangular window of \mathchoice{\mbox{\bf\displaystyle\tt x}}{\mbox{\bf\textstyle\tt x}}{\mbox{\bf\scriptstyle\tt x}}{\mbox{\bf\scriptscriptstyle\tt x}}_{\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}} = , \mathchoice{\mbox{\bf\displaystyle\tt y}}{\mbox{\bf\textstyle\tt y}}{\mbox{\bf\scriptstyle\tt y}}{\mbox{\bf\scriptscriptstyle\tt y}}_{\mathchoice{\mbox{\bf\displaystyle\tt w}}{\mbox{\bf\textstyle\tt w}}{\mbox{\bf\scriptstyle\tt w}}{\mbox{\bf\scriptscriptstyle\tt w}}} = across 313 frames, same as used in Kumar et al. work [34]. Fig.(5) shows the reconstruction results on these sequence with comparative results provided in Table (1).
3. Results on Actor dataset: Beeler et al. [7] introduced Actor dataset for high-quality facial performance capture. This dataset is composed of 346 frames captured from seven cameras with 1,180,232 vertices. The dataset captures the fine details of facial expressions which is extremely useful in the testing of NRSfM algorithms. Nevertheless, for our experiment, we require dense 2D image feature correspondences across all images as input, which we synthesized using ground-truth 3D points and synthetically generated orthographic camera rotations. To maintain the consistency with the previous works in dense NRSfM for performance evaluations, we synthesized two different datasets namely Actor Sequence1 and Actor Sequence2 based on the head movement as described in Ansari et al. work [4]. Fig.(6) shows the dense detailed reconstruction that is achieved using our algorithm. Table (1) clearly indicates the benefit of our approach to reconstruct such complex deformations.
4. Results on Face, Heart, Back dataset: To evaluate the variational approach to dense NRSfM [23] Garg et al. introduced these datasets. Its sequences are composed of monocular video’s captured in a natural environment with varying lighting condition and large displacements. It consists of three different videos with 120, 150 and 80 frames for face sequence, back sequence and heart sequence respectively. Additionally, it provides dense 2D feature track for the same with 28332, 20561, and 68295 features track over the frames for face, back and heart sequence. No ground-truth 3D is available with this dataset for evaluation. Fig.(5) show reconstruction results on back and heart sequence. For more qualitative results on these sequences, kindly refer to the supplementary material.
6.1 Algorithmic Analysis
We performed some more experiments to understand the behavior of our algorithm under different input parameters and evaluation setups. In practice these experiment help analyze the practical applicability of our algorithm.
1. Performance over noisy trajectories: We utilized the standard experimental procedure to analyze the behavior of our algorithm under different noise levels. Similar to the work of Kumar et al. [34], we added the Gaussian noise to the input trajectories. The standard deviation of the noise are adjusted as \sigma_{g}=\lambda_{g}\textrm{max}\{|\mathchoice{\mbox{\bf\displaystyle\tt W}}{\mbox{\bf\textstyle\tt W}}{\mbox{\bf\scriptstyle\tt W}}{\mbox{\bf\scriptscriptstyle\tt W}}|\} with varying from 0.01 to 0.055. Fig.(7(a)) show the quantitative comparison of our approach with recent algorithms namely DS [17] and SDG [34]. The graph is plotted by taking the average reconstruction error of all the four synthetic face dataset [23]. The procured statistics indicate that our algorithm is more resilient to noise than other competing methods.
2. Performance with change in the number of singular values: The selection of ‘’ in \mathcal{G}(\mathchoice{\mbox{\bf\displaystyle\tt p}}{\mbox{\bf\textstyle\tt p}}{\mbox{\bf\scriptstyle\tt p}}{\mbox{\bf\scriptscriptstyle\tt p}},\mathchoice{\mbox{\bf\displaystyle\tt d}}{\mbox{\bf\textstyle\tt d}}{\mbox{\bf\scriptstyle\tt d}}{\mbox{\bf\scriptscriptstyle\tt d}}) i.e. the number of top singular vectors for Grassmannian representation and its corresponding singular values to perform reconstruction can directly affect the performance of our algorithm. However, it has been observed over several experiments that we need very few singular value and singular vectors to recover dense detailed 3D reconstruction of the deforming object. Fig.(7(b)) show the variation in average 3D reconstruction with the values of ‘’ for synthetic face dataset [23].
3. Processing Time and Convergence: Our algorithm execution time is almost at par or a bit slower than SDG [34]. Fig.(7(c)) show the processing time taken by our method on different datasets. Fig.(7(d)) show a typical convergence curve of our algorithm. Ideally, it takes 120-150 iteration to provide an optimal solution to the problem.
7 Conclusion
Our Grassmannian representation of a non-rigidly deforming surface exploits the advantage of Grassmannians of different dimensions to jointly estimate better grouping of subspaces and their corresponding 3D geometry. Our approach explicitly leverages the geometric structure of the non-rigidly moving object w.r.t its neighbors on manifold via similarity graph and, it’s embedding in the lower dimension. We empirically demonstrated that our method is able to achieve 3D reconstruction accuracy which is better or as good as the state-of-the-art, with significant improvement in handling noisy trajectories.
Acknowledgement. The author was supported in part by the ARC Centre of Excellence for Robotic Vision (CE140100016), ARC Discovery project on 3D computer vision for geo-spatial localisation (DP190102261) and ARC DECRA project (DE140100180). The author thank his elder brother Aditya Kumar for his relentless support and advise. Thank you Roya Safaei for her constant help and constructive suggestions. Also, the author thank Dr. J. Varela and Dr. C. Punch, ACT for the medical treatment during the course of this work.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds . Princeton University Press, 2009.
- 2[2] Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Nonrigid structure from motion in trajectory space. In Advances in neural information processing systems , pages 41–48, 2009.
- 3[3] Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(7):1442–1456, 2011.
- 4[4] Mohammad Dawud Ansari, Vladislav Golyanik, and Didier Stricker. Scalable dense monocular surface reconstruction. ar Xiv preprint ar Xiv:1710.06130 , 2017.
- 5[5] David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms , pages 1027–1035. Society for Industrial and Applied Mathematics, 2007.
- 6[6] Jonathan F Bard. Practical bilevel optimization: algorithms and applications , volume 30. Springer Science & Business Media, 2013.
- 7[7] Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W Sumner, and Markus Gross. High-quality passive facial performance capture using anchor frames. In ACM Transactions on Graphics (TOG) , volume 30, page 75. ACM, 2011.
- 8[8] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning , 3(1):1–122, 2011.