Temporally Coherent General Dynamic Scene Reconstruction

TL;DR

This paper presents a novel method for 4D dynamic scene reconstruction from multiple wide-baseline cameras, achieving improved accuracy and temporal coherence without prior scene knowledge, enabling applications like free-viewpoint rendering.

Contribution

It introduces an automatic coarse reconstruction, temporal correspondence with joint segmentation, and a robust shape-constrained refinement approach for dynamic scenes.

Findings

Improved multi-view segmentation accuracy

Enhanced dense reconstruction quality

Effective 4D scene modeling for VR applications

Abstract

Existing techniques for dynamic scene reconstruction from multiple wide-baseline cameras primarily focus on reconstruction in controlled environments, with fixed calibrated cameras and strong prior constraints. This paper introduces a general approach to obtain a 4D representation of complex dynamic scenes from multi-view wide-baseline static or moving cameras without prior knowledge of the scene structure, appearance, or illumination. Contributions of the work are: An automatic method for initial coarse reconstruction to initialize joint estimation; Sparse-to-dense temporal correspondence integrated with joint multi-view segmentation and reconstruction to introduce temporal coherence; and a general robust approach for joint segmentation refinement and dense reconstruction of dynamic scenes by introducing shape constraint. Comparison with state-of-the-art approaches on a variety of complex indoor and outdoor scenes, demonstrates improved accuracy in both multi-view segmentation and dense reconstruction. This paper demonstrates unsupervised reconstruction of complete temporally coherent 4D scene models with improved non-rigid object segmentation and shape reconstruction and its application to free-viewpoint rendering and virtual reality.

Figures

Tables

Table 1. Table 1: Properties of all datasets where Type represents whether the data is static or dynamic. In No. of views S stands for static cameras and M for moving cameras.

Datasets	Resolution	No. of views	Baseline	Type
Officecvssp3d	$1920\times 1080$	8(all S)	$25°$-$35°$	dynamic
JugglerUnstructuredVBR10	$960\times 544$	6(all M)	$15°$-$45°$	dynamic
Dance1cvssp3d	$1920\times 1080$	8(all S)	$20°$-$30°$	dynamic
Odzemokcvssp3d	$1920\times 1080$	8(2 M)	$15°$-$30°$	dynamic
Dance24DInria	$780\times 582$	8(all S)	$35°$-$45°$	dynamic
MagicianUnstructuredVBR10	$960\times 544$	6(all M)	$15°$-$45°$	dynamic
CouchKowdle12	$640\times 480$	9(all S)	$25°$-$30°$	static
ChairKowdle12	$640\times 480$	17(all S)	$5°$-$8°$	static
CarKowdle12	$640\times 480$	16(all S)	$5°$-$8°$	static

Table 2. Table 2: Parameters used for all datasets: λ c subscript 𝜆 𝑐 \lambda_{c} represents λ c ​ o ​ n ​ t ​ r ​ a ​ s ​ t subscript 𝜆 𝑐 𝑜 𝑛 𝑡 𝑟 𝑎 𝑠 𝑡 \lambda_{contrast}

	${\lambda }_{data}$	${\lambda }_c$	${\lambda }_{smooth}$	${\lambda }_{color}$
Magician/Dance2	0.4	5.0	.0005	0.6
Juggler	0.5	5.0	.0005	0.4
Odzemok/Dance1/Office	0.4	3.0	.001	0.6

Table 3. Table 3: Static segmentation completeness comparison with existing methods on benchmark datasets ( % percent \% )

Dataset	Kowdle	Djelouah	Guillemaut	Mustafa	Proposed
Couch	$99.6\pm 0.1$	$99.0\pm 0.2$	$97.0\pm 0.3$	$98.5\pm 0.2$	99.7$\pm$0.3
Chair	99.2$\pm$0.4	$98.6\pm 0.3$	$97.9\pm 0.5$	$98.0\pm 0.5$	$99.1\pm 0.3$
Car	$98.0\pm 0.7$	$97.0\pm 0.8$	$95.0\pm 0.7$	$97.6\pm 0.3$	98.6$\pm$0.4

Table 4. Table 4: Dynamic scene segmentation completeness ( % percent \% ): P s ​ m ​ o ​ o ​ t ​ h = P ​ r ​ o ​ p ​ o ​ s ​ e ​ d − E s ​ m ​ o ​ o ​ t ​ h subscript 𝑃 𝑠 𝑚 𝑜 𝑜 𝑡 ℎ 𝑃 𝑟 𝑜 𝑝 𝑜 𝑠 𝑒 𝑑 subscript 𝐸 𝑠 𝑚 𝑜 𝑜 𝑡 ℎ P_{smooth}=Proposed-E_{smooth} , P c ​ o ​ n ​ t ​ r ​ a ​ s ​ t = P ​ r ​ o ​ p ​ o ​ s ​ e ​ d − E c ​ o ​ n ​ t ​ r ​ a ​ s ​ t subscript 𝑃 𝑐 𝑜 𝑛 𝑡 𝑟 𝑎 𝑠 𝑡 𝑃 𝑟 𝑜 𝑝 𝑜 𝑠 𝑒 𝑑 subscript 𝐸 𝑐 𝑜 𝑛 𝑡 𝑟 𝑎 𝑠 𝑡 P_{contrast}=Proposed-E_{contrast} , P d ​ s = P ​ r ​ o ​ p ​ o ​ s ​ e ​ d − E d ​ a ​ t ​ a − E s ​ m ​ o ​ o ​ t ​ h subscript 𝑃 𝑑 𝑠 𝑃 𝑟 𝑜 𝑝 𝑜 𝑠 𝑒 𝑑 subscript 𝐸 𝑑 𝑎 𝑡 𝑎 subscript 𝐸 𝑠 𝑚 𝑜 𝑜 𝑡 ℎ P_{ds}=Proposed-E_{data}-E_{smooth}

	Magician	Juggler	Odzemok	Dance1	Office	Dance2
Guillemaut	$68.0$	$84.6$	$90.1$	$99.2$	$99.3$	$98.6$
Mustafa	$88.7$	$87.9$	$89.9$	$99.4$	$99.0$	$99.0$
$P_{contrast}$	$79.6$	$88.9$	$90.5$	$99.1$	$99.2$	$98.8$
$P_{smooth}$	$89.9$	$90.6$	$90.8$	$99.2$	$99.2$	$98.9$
$P_{ds}$	$78.3$	$86.2$	$90.0$	$99.3$	$99.1$	$98.7$
Proposed	91.2	93.3	91.8	99.5	99.4	99.0

Table 5. Table 5: Comparison of computational efficiency for dynamic datasets (time in seconds (s))

Dataset	Furukawa	Guillemaut	Mustafa	Ours
Dance1	326	493	295	254
Magician	311	608	377	325
Odzemok	381	598	394	363
Office	339	533	347	291
Juggler	394	634	411	378
Dance2	312	432	323	278

Equations

$$\Vert H_{t,v}(p)+u_{t,r}(p+H_{t,v}(p))-u_{t,v}(p)-$$

$$H_{t,r}(p+u_{t,v}(p))\Vert <ϵ$$

$${\lambda }_{smooth}{E}_{smooth}(l,d)+{\lambda }_{color}{E}_{color}(l)\text{vspace}-0.2cm$$

$$E^{\star }(l|x,c)=\sum _{p\in R}\sum _{q\in {\Gamma }_{c,p}}{E}_{p,q}^{\star }(l_p,l_q)$$

$$\forall q\in {\Gamma }_{c,p},E_{p,q}^{\star }=\{\begin{array}{c}\infty \text{ if }{l}_p\ne l_q\\ 0\text{ otherwise }\end{array}$$

$${\Gamma }_{c,p}=\arg \underset{\Gamma \in P_{c,p}}{\min }L(\Gamma )$$

$$L(\Gamma )=\sum _{i=1}^{N_D-1}\sqrt{(1-{\delta }_g)j({\Gamma }^i,{\Gamma }^{i+1}{)}^2+{\delta }_g{\Vert ▽I({\Gamma }^i)\Vert }^2}$$

$$E^{\star }(l|x,\mathcal{C})=\sum _{p\in R}\sum _{q\in {\Gamma }_{c,p}}{E}_{p,q}^{\star }(l_p,l_q)$$

$$\underset{s.t.}{mi{n}_{(l,d)}}\underset{lϵS^{\star }(\mathcal{C})}{E(l,d)}\iff \underset{(l,d)}{\min }E(l,d)+E^{\star }(l|x,\mathcal{C})$$

$$\{\begin{array}{ll}M(p,q)={\sum }_{i\in {\mathcal{O}}_k}m(p,q),& \text{if }{d}_p\ne \mathcal{U}\\ M_{\mathcal{U}},& \text{if }{d}_p=\mathcal{U}\end{array}\text{vspace}-0.5cm$$

$$m(p,q)=\frac{exp\frac{c_{min}}{2{\sigma }_i^2}}{{\sum }_{(p,q)\in \mathcal{N}}exp\frac{-(1-NCC(p,q))}{2{\sigma }_i^2}}\text{vspace}-0.2cm$$

$$E_{contrast}(l)=\sum _{p,q\in \mathcal{N}}{e}_{contrast}(p,q,l_p,l_q)$$

$$e_{contrast}(p,q,l_p,l_q)=\{\begin{array}{ll}0,& \text{if }(l_p=l_q)\\ \frac{1}{1+ϵ}(ϵ+ex{p}^{-C(p,q)}),& \text{otherwise}\end{array}$$

$$E_{smooth}(l,d)=\sum _{(p,q)\in \mathcal{N}}{e}_{smooth}(l_p,d_p,l_q,d_q)=\text{vspace}-0.2cm$$

$$\{\begin{array}{ll}min(\left|d_p-d_q\right|,d_{max}),& \text{if }{l}_p=l_q\text{ and }{d}_p,d_q\ne \mathcal{U}\\ 0,& \text{if }{l}_p=l_q\text{ and }{d}_p,d_q=\mathcal{U}\\ d_{max},& \text{otherwise}\text{vspace}-0.2cm\end{array}$$

$$E_{color}(l)=\sum _{p\in \mathcal{P}}-logP(I_p|l_p)\text{vspace}-0.5cm$$

Full text

††journal: International Journal of Computer Vision

1 Introduction

Reconstruction of general dynamic scenes is of great importance in entertainment applications such as visual effects in film and broadcast production and for content production in virtual reality. The ultimate goal of modelling dynamic scenes from multiple cameras is automatic understanding of real-world scenes from distributed camera networks, for applications in robotics and other autonomous systems. Existing multi-view dynamic scene reconstruction methods either work in controlled environment with known background or chroma-key studio Guillemaut2010 ; Goldluecke04 ; Starck2007 ; taneja2011modeling or require a large number of cameras Furukawa2010 . Extensions to more general outdoor scenes UnstructuredVBR10 ; Kim2012 ; taneja2011modeling use prior reconstruction of the static geometry from images of the empty environment. However these methods either require accurate segmentation of dynamic foreground objects, or prior knowledge of the scene structure and background, or are limited to static cameras and controlled environments. Scenes are reconstructed semi-automatically, requiring manual intervention for segmentation/rotoscoping, and result in temporally incoherent per-frame mesh geometries. Temporally coherent geometry with known surface correspondence across the sequence is essential for real-world applications and compact representation.

This paper addresses the limitations of existing approaches by introducing a methodology for unsupervised 4D temporally coherent dynamic scene reconstruction from multiple wide-baseline static or moving camera views without prior knowledge of the scene structure or background appearance. The scene is automatically decomposed into a set of spatio-temporally coherent objects as shown in Figure 1 where the resulting 4D scene reconstruction has temporally coherent labels and surface correspondence for each object. This temporally coherent dynamic scene reconstruction is demonstrated to work in applications for immersive content production such as free-viewpoint video (FVV) and virtual reality (VR). The contributions are summarized as follows:

• •

Unsupervised temporally coherent dense reconstruction and segmentation of general complex dynamic scenes from multiple wide-baseline views.

• •

Automatic initialization of dynamic object segmentation and reconstruction from sparse features.

• •

A framework for space-time sparse-to-dense segmentation, reconstruction and temporal correspondence.

• •

Robust spatio-temporal refinement of dense reconstruction and segmentation integrating error tolerant photo-consistency and edge information using geodesic star convexity.

• •

Robust and computationally efficient reconstruction of dynamic scenes by exploiting temporal coherence.

• •

Real-world applications of 4D reconstruction to free-viewpoint video rendering and virtual reality.

This paper presents a unified framework from two previously published papers, combining multiple view joint reconstruction and segmentation MustafaICCV15 with temporal coherence Mustafa16 to improve per-frame reconstruction performance. In particular the approach estimates a 4D surface model with full correspondence over time. A comprehensive experimental evaluation with comparison to the state-of-the-art in segmentation, reconstruction and 4D modelling is also presented extending previous work. Application of the resulting 4D models to FVV rendering and content production for immersive VR experiences is also presented.

2 Related Work

Temporally coherent reconstruction is a challenging task for general dynamic scenes due to a number of factors such as motion blur, articulated, non-rigid and large motion of multiple people, resolution differences between camera views, occlusions, wide-baselines, errors in calibration and cluttered dynamic backgrounds. Segmentation of dynamic objects from such scenes is difficult because of foreground and background complexity and the likelihood of overlapping background and foreground color distributions. Reconstruction is also challenging due to limited visual cues and relatively large errors affecting both calibration and extraction of a globally consistent solution. This section reviews previous work on dynamic scene reconstruction and segmentation.

2.1 Dynamic Scene Reconstruction

Dense dynamic shape reconstruction is a fundamental problem and heavily studied area in the field of computer vision. Recovering accurate 3D models of a dynamically evolving, non-rigid scene observed by multiple synchronised cameras is a challenging task. Research on multiple view dense dynamic reconstruction has primarily focused on indoor scenes with controlled illumination and static backgrounds, extending methods for multiple view reconstruction of static scenes Seitz06 to sequences Tung09 . Deep learning based approaches have been introduced to estimate shape of dynamic objects from minimal camera views in constrained environment Huang_2018_ECCV ; Wu_2018_ECCV and for rigid objects Stutz2018 . In the last decade, focus has shifted to more challenging outdoor scenes captured with both static and moving cameras. Reconstruction of non-rigid dynamic objects in uncontrolled natural environments is challenging due to the scene complexity, illumination changes, shadows, occlusion and dynamic backgrounds with clutter such as trees or people. Methods have been proposed for multi-view reconstruction Vo_2016_CVPR ; Cheng09 ; Larsen07 requiring a large number of closely spaced cameras for surface estimation of dynamic shape. Practical applications require relatively sparse moving cameras to acquire coverage over large areas such as outdoor. A number of approaches for mutli-view reconstruction of outdoor scenes require initial silhouette segmentation Wu2013 ; Kim2012 ; Li10 ; Guillemaut2010 to allow visual-hull reconstruction. Most of these approaches to general dynamic scene reconstruction fail in the case of complex (cluttered) scenes captured with moving cameras.

A recent work proposed reconstruction of dynamic fluids Qian_2017_CVPR for static cameras. Another work used RGB-D cameras to obtain reconstruction of non-rigid surfaces Slavcheva_2017_CVPR . Pioneering research in general dynamic scene reconstruction from multiple handheld wide-baseline cameras UnstructuredVBR10 ; taneja2011modeling exploited prior reconstruction of the background scene to allow dynamic foreground segmentation and reconstruction. Recent work Ngo2019 estimates shape of dynamic objects from handheld cameras exploiting GANs. However these approaches either work for static/indoor scenes or exploit strong prior assumptions such as silhouette information, known background or scene structure. Also all these approaches give per frame reconstruction leading to temporally incoherent geometries. Our aim is to perform temporally coherent dense reconstruction of unknown dynamic non-rigid scenes automatically without strong priors or limitations on scene structure.

2.2 Multi-view Video Segmentation

In the field of image segmentation, approaches have been proposed to provide temporally consistent monocular video segmentation GrundmannKwatra2010 ; Papazoglou13 ; Narayana13 ; Zhang13 . Hierarchical segmentation based on graphs was proposed in GrundmannKwatra2010 , directed acyclic graph was used to propose an object segmentation Zhang13 and optical flow is used to consistently segment objects Narayana13 ; Papazoglou13 . Recently a number of approaches have been proposed for multi-view foreground object segmentation by exploiting appearance similarity spatially across views Djelouah13 ; Kowdle12 ; Lee11 ; Zeng04 and space-time similarity Djelouah15 . However, multi-view approaches assume a static background and different colour distributions for the foreground and background which limits applicability for general scenes and non-rigid objects.

To address this issue we introduce a novel method for spatio-temporal multi-view segmentation of dynamic scenes using shape constraints. Single image segmentation techniques using shape constraints provide good results for complex scenes Gulshan10 (convex and concave shapes), but require manual interaction. The proposed approach performs automatic multi-view video segmentation by initializing the foreground object model using spatio-temporal information from wide-baseline feature correspondence followed by a multi-layer optimization framework. Geodesic star convexity previously used in single view segmentation Gulshan10 is applied to constraint the segmentation in each view. Our multi-view formulation naturally enforces coherent segmentation between views and resolves ambiguities such as the similarity of background/foreground in isolated views.

2.3 Joint Segmentation and Reconstruction

Joint segmentation and reconstruction methods incorporate estimation of segmentation or matting with reconstruction to provide a combined solution. Joint refinement avoids the propagation of errors between the two stages thereby making the solution more robust. Also, cues from segmentation and reconstruction can be combined efficiently to achieve more accurate results. The first multi-view joint estimation system was proposed by Szeliski et al.Szeliski99 which used iterative gradient descent to perform an energy minimization. A number of approaches were introduced for joint formulation in static scenes and one recent work used training data to classify the segments Zach13joint3d . The focus shifted to joint segmentation and reconstruction for rigid objects in indoor and outdoor environments. These approaches used a variety of techniques such as patch-based refinement Shin2013 ; Ozden07 and fixating cameras on the object of interest Campbell201014 for reconstructing rigid objects in the scene. However, these are either limited to static scenes Zach13joint3d ; Hane13 or process each frame independently thereby failing to enforce temporal consistency Campbell201014 ; Guillemaut2010 .

Joint reconstruction and segmentation on monocular video was proposed in Abhijit14 ; Abarghouei_2019_CVPR ; Chen_2019_CVPR achieving semantic segmentation of scene limited to rigid objects in street scenes. Practical application of joint estimation requires these approaches to work on non-rigid objects such as humans with clothing. A multi-layer joint segmentation and reconstruction approach was proposed for multiple view video of sports and indoor scenes Guillemaut2010 . The algorithm used known background images of the scene without the dynamic foreground objects to obtain an initial segmentation. Visual-hull based reconstruction was performed with known prior foreground/background using a background image plate with fixed and calibrated cameras. This visual hull was used as a prior and was optimized by a combination of photo-consistency, silhouette, color and sparse feature information in an energy minimization framework to improve the segmentation and reconstruction quality. Although structurally similar to our approach, it requires the scene to be captured by fixed calibrated cameras and a priori known fixed background plate as a prior to estimate the initial visual hull by background subtraction. The proposed approach overcomes these limitations allowing moving cameras and unknown scene backgrounds. These methods are able to produce high quality results, but rely on good initializations and strong prior assumptions with known and controlled (static) scene backgrounds.

To overcome the limitations of existing methods, the proposed approach automatically initialises the foreground object segmentation from wide-baseline correspondence without prior knowledge of the scene. This is followed by a joint spatio-temporal reconstruction and segmentation of general scenes.

2.4 Temporally coherent 4D Reconstruction

Temporally coherent 4D reconstruction refers to aligning the 3D surfaces of non-rigid objects in time for a dynamic sequence. This is achieved by estimating point-to-point correspondences for the 3D surfaces to obtain temporally coherent 4D reconstruction. 4D models allows to create efficient representation for practical applications in film, broadcast and immersive content production such as virtual, augmented and mixed reality. The existing approaches for multi-view reconstruction of dynamic scenes process each time frame independently. Independent per-frame reconstruction can result in errors due to the inherent visual ambiguity caused by occlusion and similar object appearance for general scenes.

3D scene flow Menze2015CVPR estimates frame-to-frame correspondence or exploits 2D optical flow Wedel2011 ; Basha2013 . These methods require an accurate estimate for most of the pixels which fails in the case of large motion. However, 3D scene flow methods align two frames independently and do not produce temporally coherent 4D models across the complete sequence to obtain a single surface model. Research investigating spatio-temporal reconstruction across multiple frames was proposed by Goldluecke04 ; Larsen07 ; Guillemaut3dv ; Mustafa16ECCV exploiting the temporal information from the previous frames using optical flow. An approach for recovering space-time consistent depth maps from multiple video sequences captured by stationary, synchronized and calibrated cameras for depth based free viewpoint video rendering was proposed by Cheng09 . However these methods require accurate initialisation, fixed and calibrated cameras and are limited to simple scenes. Other approaches to temporally coherent reconstruction either require a large number of closely spaced cameras Bailer15 or bi-layer segmentation Zhang11robustbilayer ; Jiang12 as a constraint for reconstruction. Recent approaches for spatio-temporal reconstruction of multi-view data work on indoor data Oswald14 .

The proposed framework addresses limitations of existing approaches and gives 4D temporally coherent reconstruction for general dynamic indoor or outdoor scenes with large non-rigid motions, repetitive texture, uncontrolled illumination, and large capture volume. The proposed approach gives 4D models of complete scenes with both static and dynamic objects for real-world applications (FVV and VR) with no prior knowledge of scene structure.

2.5 Summary and Motivation

Image-based temporally coherent 4D dynamic scene reconstruction without constraints is a key problem in computer vision. Existing dense reconstruction algorithms need some strong initial prior and constraints for the solution to converge such as background, structure, and segmentation, which limits their application for automatic reconstruction of general scenes. Current approaches are also commonly limited to independent per-frame reconstruction and do not exploit temporal information or produce a 4D model with known correspondence. The approach proposed in this paper aims to overcome the limitations of existing approaches by initializing the joint reconstruction and segmentation algorithm automatically, introducing temporal coherence in the reconstruction and geodesic star convexity in segmentation to reduce ambiguity and ensure consistent non-rigid structure initialization at successive frames.

3 Methodology

An overview of the proposed framework for temporally coherent multi-view reconstruction is presented in Figure 2 and consists of the following stages:

Multi-view video: The scenes are captured using multiple video cameras (static/moving) separated by wide-baseline ($>15^{\circ}$). Cameras can be synchronised either directly or in post-processing using the audio information. The cameras can be synchronized using time-code generator or later using the audio information. Camera intrinsics are known. Camera extrinsics (location,orientation) and scene structure are assumed to be unknown.

Sparse reconstruction: Segmentation based feature detection (SFD) Mustafa15 ; Mustafa2019MSFDMS is used to detect sparse features distributed throughout the scene for wide-baseline matching. SFD features are matched between views using a SIFT descriptor. The camera extrinsics and sparse 3D feature locations are then estimated for each time instant for the entire sequence Hartley03 .

Initial coarse reconstruction - Section 3.1: The sparse point cloud is clustered in 3D RusuDoctoralDissertation with each cluster representing a unique foreground object. Automatic initialisation is performed without prior knowledge of the scene structure or appearance to obtain an initial approximation for each object.

Sparse-to-dense temporal reconstruction - Section 3.2 Temporal coherence is introduced to initialize the coarse reconstruction and obtain frame-to-frame dense correspondences for dynamic objects. Sparse temporal correspondence helps in identifying dynamic objects and allows propagation of the dense reconstruction between time instants to obtain an initialization.

Joint refinement of shape and segmentation - Section 3.3: The initial estimate is refined for each object per-view through joint optimisation of shape and segmentation using a robust cost function combining matching, color, contrast and smoothness information with a geodesic star convexity constraint. A single 3D model for each dynamic object is obtained by fusion of the view-dependent depth maps using Poisson surface reconstruction Kazhdan2006 . Surface orientation is estimated based on neighbouring pixels.

3.1 Initial Coarse Reconstruction

For general dynamic scene reconstruction, we need to reconstruct and segment the objects in the scene. This requires an initial coarse approximation for initialisation of a subsequent refinement step to optimise the segmentation and reconstruction. Sparse point cloud clustering is used to segment objects, an overview is shown in Figure 3. The dense reconstruction of the foreground objects and background are combined to obtain a full scene reconstruction at the first time instant. A coarse geometric proxy of the background is created. For consecutive time instants dynamic objects and newly appeared objects are identified and only these objects are reconstructed and segmented. The reconstruction of static objects is retained which reduces computational complexity. The optic flow and cluster information for each dynamic object ensures that we retain consistent labels for the entire sequence.

3.1.1 Background Reconstruction

Accurate reconstruction of the background is often challenging due to uniform appearance of large regions. A coarse geometric proxy of the background is created by computing the minimum oriented bounding box for the sparse 3D point-cloud using principal component analysis Dimitrov06 . Different methods are used for background estimation for indoor and outdoor scenes. For outdoor scenes a plane is inserted at infinity perpendicular to the ground plane as there are no consistent constraints. For indoor scenes the Manhattan world assumption CoughlanY00 is applied to estimate room structure. The process used for estimation of the background is described below:

• •

The centroid A $=(a_{0},a_{1},a_{2})$ and normalized covariance of the point-cloud are estimated to compute the eigenvectors $\vv{e}=(e_{0},e_{1},e_{2})$ for the covariance matrix of the point-cloud. We define the reference system as R$=(e_{0},e_{1},e_{0}\times e_{1})$ such that: $e_{0}\times e_{1}=\pm e_{2}$. Rotation matrix R and translation A are used to map sparse points in the first frame and place a box in correct location.

• •

The minimum and maximum values of coordinates in the x, y and z directions for the transformed cloud are computed to determine the minimum oriented box width, height, and depth.

• •

Given a box centred at the origin with size defined above the rotation R and translation R$\times$C$+$A is applied, where C is the middle of the minimum and maximum points.

This background reconstruction is a rough geometric proxy estimate of the background of the scene but gives reasonable results for complete scene reconstruction.

3.1.2 Sparse Point-cloud Clustering

The sparse representation of the scene is processed to remove outliers using the point neighbourhood statistics to filter outlier data RusuDoctoralDissertation . We segment the objects in the sparse scene reconstruction, this allows only moving objects to be reconstructed at each frame for efficiency and this also allows object shape similarity to be propagated across frames to increase robustness of reconstruction. Object segmentation increases efficiency and improve robustness of 4D models.

We use data clustering approach based on the 3D grid subdivision of the space using an octree data structure in Euclidean space to segment objects at each frame RusuDoctoralDissertation . In a more general sense, nearest neighbor information is used to cluster, which is essentially similar to a flood fill algorithm. We choose this data clustering because of its computational efficiency and robustness. The approach allows segmentation of objects in the scene and is demonstrated to work well for cluttered and general outdoor scenes as shown in Section 4.

Objects with insufficient detected features are reconstructed as part of the scene background. Appearing, disappearing and reappearing objects are handled by sparse dynamic feature tracking, explained in Section 3.2. Clustering results are shown in Figure 3. This is followed by a sparse-to-dense coarse object based approach to segment and reconstruct general dynamic scenes.

3.1.3 Coarse Object Reconstruction

The process to obtain the coarse reconstruction for the first frame of the sequence is shown in Figure 4. The sparse representation of each element is back-projected on the rectified image pair for each view. Delaunay triangulation Fortune97 is performed on the set of back projected points for each cluster on one image and is propagated to the second image using the sparse matched features. Triangles with edge length greater than the median length of edges of all triangles are removed. For each remaining triangle pair direct linear transform is used to estimate the affine homography. Displacement at each pixel within the triangle pair is estimated by interpolation to get an initial dense disparity map for each cluster in the 2D image pair labelled as $\mathscr{R}_{I}$ depicted in red in Figure 4. The initial coarse reconstruction for the observed objects in the scene is used to define the depth hypotheses at each pixel for the optimization.

The region $\mathscr{R}_{I}$ does not ensure complete coverage of the object, so we extrapolate this region to obtain a region $\mathscr{R}_{O}$ (shown in yellow) in 2D by $5\%$ of the average distance between the boundary points($\mathscr{R}_{I}$) and the centroid of the object. To allow for errors in the initial approximate depth from sparse features we add volume in front and behind of the projected surface by an error tolerance, along the optical ray of the camera. This ensures that the object boundaries lie within the extrapolated initial coarse estimate. The tolerance for extrapolation may vary if a pixel belongs to $\mathscr{R}_{I}$ or $\mathscr{R}_{O}$ as the propagated pixels of the extrapolated regions ($\mathscr{R}_{O}$) may have a high level of error compared to error at the points from sparse representation ($\mathscr{R}_{I}$) requiring a comparatively higher tolerance. The calculation of threshold depends on the capture volume of the datasets and is set to $1\%$ of the capture volume for $\mathscr{R}_{O}$ and half the value for $\mathscr{R}_{I}$. This volume in 3D corresponds to our initial coarse reconstruction of each object and enables us to remove the dependency of previous approaches on static background plate and visual hull estimates. This process of cluster identification and initial coarse object reconstruction is performed for multiple objects in general environments. Initial object segmentation using point cloud clustering and coarse segmentation is insensitive to parameters. Throughout this work the same parameters are used for all datasets. The result of this process is a coarse initial object segmentation and reconstruction for each object.

3.2 Sparse-to-dense temporal reconstruction

Once the static scene reconstruction is obtained for the first frame, we perform temporally coherent reconstruction for dynamic objects at successive time instants instead of whole scene reconstruction for computational efficiency and to avoid redundancy. Dynamic objects are identified from the temporal correspondence of sparse feature points (Section 3.2.1), shown in Figure 5. Sparse correspondence is used to propagate and obtain an initial model of the moving object for refinement (Section 3.2.2). The initial coarse reconstruction for each dynamic region is refined in the subsequent optimization step with respect to each camera view.

3.2.1 Sparse temporal dynamic feature tracking

Numerous approaches have been proposed to track moving objects in 2D using either features or optical flow. However these methods may fail in the case of occlusion, movement parallel to the view direction, large motions and moving cameras. To overcome these limitations we match the sparse 3D feature points obtained using SFD Mustafa15 ; Mustafa2019MSFDMS from multiple wide-baseline views at each time instant. The use of sparse 3D features is robust to large non-rigid motion, occlusions and camera movement. SFD detects sparse features which are stable across wide-baseline views and consecutive time instants for a moving camera and dynamic scene. Sparse 3D feature matches between consecutive time instants are back-projected to each view. These features are matched temporally using SIFT descriptor to identify the corresponding moving points. Robust matching is achieved by enforcing multiple view consistency for the temporal feature correspondence in each view as illustrated in Figure 6. Each match must satisfy the constraint:

[TABLE]

where $p$ is the feature image point in view $v$ at frame $t$, $H_{t,v}(p)$ is the disparity at frame $t$ from views $v$ and $r$, $u_{t,v}(p)$ is the temporal correspondence from frames $t$ to $t+1$ for view $v$. The multi-view consistency check ensures that correspondences between any two views remain temporally consistent for successive frames. Matches in the 2D domain are sensitive to camera movement and occlusion, hence we map the set of refined matches into 3D to make the system robust to camera motion. The Frobenius norm is applied on the 3D point gradients in all directions Zhang13 to obtain the ‘net’ motion at each sparse point. The ‘net’ motions between pairs of 3D points for consecutive time instants are ranked, and the top and bottom $5\%$ values are removed followed by Median filtering to identify the dynamic features. New objects are identified per frame from the clustered sparse reconstruction and are labelled as dynamic objects.

3.2.2 Sparse-to-dense model reconstruction

Dynamic 3D feature points are used to initialize the segmentation and reconstruction of the initial model. This avoids the assumption of static backgrounds and prior scene segmentation commonly used to initialise multiple view reconstruction with a coarse visual-hull approximation Guillemaut2010 . Temporal coherence also provides a more accurate initialisation to overcome visual ambiguities at individual frames. Figure 7 illustrates the use of temporal coherence for reconstruction initialisation and refinement. Dynamic feature correspondence is used to identify the mesh for each dynamic object. This mesh is back projected on each view to obtain the region of interest. Lucas Kanade Optical flow Bouguet00 is performed on the projected mask for each view in the temporal domain using the dynamic feature correspondences over time as initialization. Dense multi-view wide-baseline correspondences from the previous frame are propagated to the current frame using the information from the flow vectors to obtain dense points in the current frame. The matches are triangulated in 3D to obtain a refined 3D dense model of the dynamic object for the current frame.

For dynamic scenes, to allow the introduction of new objects and object parts we use information from the cluster of sparse points for each dynamic object. The cluster corresponding to the dynamic features is identified and static points are removed. This ensures that the set of new points not only contain the dynamic features but also the unprocessed points which represent new parts of the object. These points are added to the refined sparse model of the dynamic object. To handle the new objects we detect new clusters at each time instant and consider them as dynamic regions. Once we have a set of dense 3D points for each dynamic object, Poisson surface reconstruction Kazhdan2006 is performed on the set of sparse points to obtain an initial coarse model of each dynamic region $\mathscr{R}$. The depth of region $\mathscr{R}$ is refined per view for each dynamic object at a per pixel level.

The sparse-to-dense initial coarse reconstruction improves the quality of segmentation and reconstruction after the refinement. Examples of the improvement in segmentation and reconstruction for Odzemok cvssp3d and Juggler UnstructuredVBR10 datasets are shown in Figure 8. As observed limbs of the people is correctly reconstructed by using information from the previous frames in both the cases.

3.3 Joint refinement of shape and segmentation

The initial reconstruction $\mathscr{R}$ and segmentation ($\mathscr{R}$ projected in views) from dense temporal feature correspondence is refined using a joint optimization framework. View-dependent optimisation of depth is performed with respect to each camera which is robust to errors in camera calibration and initialisation. Calibration inaccuracies produce inconsistencies limiting the applicability of global reconstruction techniques which simultaneously consider all views; view-dependent techniques are more tolerant to such inaccuracies because they only use a subset of the views for reconstruction of depth from each camera view.

Our goal is to assign an accurate depth value from a set of depth values $\mathscr{D}=\left\{d_{1},...,d_{\left|\mathscr{D}\right|-1},\mathscr{U}\right\}$ and assign a layer label from a set of label values $\mathscr{L}=\left\{l_{1},...,l_{\left|\mathscr{L}\right|}\right\}$ to each pixel $p$ for the region $\mathscr{R}$ of each dynamic object. Each $d_{i}$ is obtained by sampling the optical ray from the camera and $\mathscr{U}$ is an unknown depth value to handle occlusions. This is achieved by optimisation of a joint cost function Guillemaut2010 for label (segmentation) and depth (reconstruction):

$E(l,d)=\lambda_{data}E_{data}(d)+\lambda_{contrast}E_{contrast}(l)+$

[TABLE]

where, $d$ is the depth at each pixel, $l$ is the layer label for multiple objects and the cost function terms are defined in section 3.3.2. The equation consists of four terms: the data term is for the photo-consistency scores, the smoothness term is to avoid sudden peaks in depth and maintain the consistency and the color and contrast terms are to identify the object boundaries. Data and smoothness terms are common to solve reconstruction problems Bleyer11 and the color and contrast terms are used for segmentation Kolmogorov06 . This is solved subject to a geodesic star-convexity constraint on the labels $l$.

3.3.1 Shape constraint for joint optimization

A novel shape constraint is introduced based on geodesic star convexity which has previously been shown to give improved performance in interactive image segmentation for structures with fine details (for example a person’s fingers or hair)Gulshan10 . Previous methods used shape constraints by enforcing star convexity prior to improve segmentation Veksler08 ; Vicente08 . However star-convexity constraints fail for non-rigid objects (humans), as illustrated in Figure 9. To handle complex objects the geodesic star convexity prior with multiple star centres was introduced in interactive segmentation for 2D objects Gulshan10 . The notion of connectivity was extended from Euclidean to geodesic space so that paths adapt to image data as opposed to straight Euclidean rays, thus extending visibility and reducing the number of star centers required. The union formed by multiple object seeds and geodesic paths form a geodesic forest Gulshan10 .

In this work the geodesic star-convexity shape constraint is automatically initialised for each view from the initial segmentation and sparse features to constrain the energy minimisation for joint multi-view reconstruction and segmentation. The shape constraint is based on the geodesic distance as star centres to which the object shape is restricted. This allows complex shapes to be segmented. To automatically initialize the segmentation we use the sparse temporal feature correspondence as star centers (seeds) to build a geodesic forest.

The region outside the initial coarse reconstruction of all dynamic objects is initialized as the background seed for segmentation as shown in Figure 11. The shape of the dynamic object is restricted by this geodesic distance constraint that depends on the image gradient. Comparison with existing methods for multi-view segmentation demonstrates improvements in recovery of fine detail structure as illustrated in Figure 11.

In Equation 2 a label $l$ is star convex with respect to center $c$, if every point $p\in l$ is visible to a star center $c$ via $l$ in the image $x$ which can be expressed as an energy cost:

[TABLE]

[TABLE]

where $\forall p\in{R}:p\in l\Leftrightarrow l_{p}=1$ and $\Gamma_{c,p}$ is the geodesic path joining $p$ to the star center $c$ given by:

[TABLE]

where ${P}_{c,p}$ denotes the set of all discrete paths between $c$ and $p$ and ${L}(\Gamma)$ is the length of discrete geodesic path as defined in Gulshan10 . In the case of image segmentation the gradients in the underlying image provide information to compute the discrete paths between each pixel and star centers and ${L}(\Gamma)$ is defined below:

[TABLE]

where $\Gamma$ is an arbitrary parametrized discrete path with $N_{D}$ pixels given by $\left\{\Gamma^{1},\Gamma^{2},\cdots\Gamma^{N_{D}}\right\}$, $j(\Gamma^{i},\Gamma^{i+1})$ is the Euclidean distance between successive pixels, and the quantity $\left\|\bigtriangledown I(\Gamma^{i})\right\|^{2}$ is a finite difference approximation of the image gradient between the points $\left(\Gamma^{i},\Gamma^{i+1}\right)$. The parameter weights $\delta_{g}$ the Euclidean distance with the geodesic length. Using the above definition, one can define the geodesic distance as defined in Equation 5.

An extension of single star-convexity is to use multiple stars to define a more general class of shapes. Introduction of multiple star centers reduces the path lengths and increases the visibility of small parts of objects like small limbs as shown in Figure 9. Hence Equation 3 is extended to multiple stars. A label $l$ is star convex with respect to center $c_{i}$, if every point $p\in l$ is visible to a star center $c_{i}$ in set $\mathscr{C}=\left\{c_{1},...,c_{N_{T}}\right\}$ via $l$ in the image $x$, where $N_{T}$ is the number of star centers Gulshan10 . This is expressed as an energy cost:

[TABLE]

In our case all the correct temporal sparse feature correspondences are used as star centers, hence the segmentation will include all the points which are visible to these sparse features via geodesic distances in the region $\mathscr{R}$, thereby employing the shape constraint. Since the star centers are selected automatically, the method is unsupervised. Comparison of segmentation constraint with geodesic multi-star convexity against no constraints and Euclidean multi-star convexity constraint is shown in Figure 10. The figure demonstrates the usefulness of the proposed approach with an improvement in segmentation quality on non-rigid complex objects. The energy in the Equation 2 is minimized as follows:

[TABLE]

where $S^{\star}(\mathscr{C})$ is the set of all shapes which lie within the geodesic distances with respect to the centers in $\mathscr{C}$. Optimization of Equation 7, subject to each pixel $p$ in the region $\mathscr{R}$ being at a geodesic distance $\Gamma_{c,p}$ from the star centers in the set $\mathscr{C}$, is performed using the $\alpha$-expansion algorithm for a pixel $p$ by iterating through the set of labels in $\mathscr{L}\times\mathscr{D}$ Boykov01 . Graph-cut is used to obtain a local optimum Kolmogorov04 .

The improvements in the results using geodesic star convexity in the framework is shown in Figure 11 and by using temporal coherence is shown in Figure 8. Figure 12 shows improvements using geodesic shape constraint, temporal coherence and combined proposed approach for Dance2 4DInria dataset.

3.3.2 Energy cost function for joint optimization

For completeness in this section we define each of the terms in Equation 2, these are based on previous terms used for joint optimisation over depth for each pixel introduced in MustafaICCV15 , with modification of the color matching term to improve robustness and extension to multiple labels.

Matching term: The data term for matching between views is specified as a measure of photo-consistency (Figure 13) as follows:

$E_{data}(d)=\sum_{p\in\mathscr{P}}e_{data}(p,d_{p})=$

[TABLE]

where $\mathscr{P}$ is the 4-connected neighbourhood of pixel $p$, $M_{\mathscr{U}}$ is the fixed cost of labelling a pixel unknown and $q$ denotes the projection of the hypothesised point $P$ in an auxiliary camera where $P$ is a $3D$ point along the optical ray passing through pixel $p$ located at a distance $d_{p}$ from the reference camera. $\mathscr{O}_{k}$ is the set of $k$ most photo-consistent pairs.

For textured scenes Normalized Cross Correlation (NCC) over a squared window is a common choice Seitz06 . The NCC values range from -1 to 1 which are then mapped to non-negative values by using the function $1-NCC$. A maximum likelihood measure Larry92 is used in this function for confidence value calculation between the center pixel $p$ and the other pixels $q$ and is based on the survey on confidence measures for stereo Hu12 . The measure is defined as:

[TABLE]

where $\sigma_{i}^{2}$ is the noise variance for each auxiliary camera $i$; this parameter was fixed to $0.3$. $\mathscr{N}$ denotes the set of interacting pixels in $\mathscr{P}$. $c_{min}$ is the minimum cost for a pixel obtained by evaluating the function $(1-NCC(.,.))$ on a $15\times 15$ window.

Contrast term: Segmentation boundaries in images tend to align with contours of high contrast and it is desirable to represent this as a constraint in stereo matching. A consistent interpretation of segmentation-prior and contrast-likelihood is used from Kolmogorov06 . We used a modified version of this interpretation in our formulation to preserve the edges by using Bilateral filtering Tomasi1998 instead of Gaussian filtering. The contrast term is as follows:

[TABLE]

[TABLE]

$\left\|\cdot\right\|$ is the $L_{2}$ norm and $\epsilon=1$. The simplest choice for $C(p,q)$ would be the squared Euclidean color distance between intensities at pixel $p$ and $q$ as used in Guillemaut2010 . We propose a term for better segmentation as $C(p,q)=\frac{\left\|B(p)-B(q)\right\|^{2}}{2\sigma_{pq}^{2}d_{pq}^{2}}$ where $B(.)$ represents the bilateral filter, $d_{pq}$ is the Euclidean distance between $p$ and $q$, and $\sigma_{pq}=\left\langle\frac{\left\|B(p)-B(p)\right\|^{2}}{d_{pq}^{2}}\right\rangle$ This term enables to remove the regions with low photo-consistency scores and weak edges and thereby helps in estimating the object boundaries.

Smoothness term: This term is inspired by Guillemaut2010 and it ensures the depth labels vary smoothly within the object reducing noise and peaks in the reconstructed surface. This is useful when the photo-consistency score is low and insufficient to assign depth to a pixel (Figure 13). It is defined as:

[TABLE]

[TABLE]

$d_{max}$ is set to 50 times the size of the depth sampling step for all datasets.

Color term: This term is computed using the negative log likelihood Kolmogorov04 of the color models learned from the foreground and background markers. The star centers obtained from the sparse 3D features are foreground markers and for background markers we consider the region outside the projected initial coarse reconstruction for each view. The color models use GMMs with 5 components each for Foreground/Background mixed with uniform color models Das09 as the markers are sparse.

[TABLE]

where $P(I_{p}\rvert l_{p}=l_{i})$ denotes the probability at pixel $p$ in the reference image belonging to layer $l_{i}$.

4 Results and Performance Evaluation

The proposed system is tested on publicly available multi-view research datasets of indoor and outdoor scenes, details of datasets explained in Table 1. The parameters used for all the datasets are defined in Table 2. More information is available on the website111http://cvssp.org/projects/4d/4DRecon/.

4.1 Multi-view segmentation evaluation

Segmentation is evaluated against the state-of-the-art methods for multi-view segmentation Kowdle Kowdle12 and Djelouah Djelouah13 for static scenes and joint segmentation reconstruction methods Mustafa MustafaICCV15 (per frame) and Guillemaut Guillemaut3dv (using temporal information) for both static and dynamic scenes.

For static multi-view data the segmentation is initialised as detailed in Section 3 followed by refinement using the constrained optimisation Section 3.3. For dynamic scenes the full pipeline with temporal coherence is used as detailed in 3. Ground-truth is obtained by manually labelling the foreground for Office, Dance1 and Odzemok dataset, and for other datasets ground-truth is available online. We initialize all approaches by the same proposed initial coarse reconstruction for fair comparison.

To evaluate the segmentation we measure completeness as the ratio of intersection to union with ground-truth Kowdle12 . Comparisons are shown in Table 3 and Figure 14, 15 for static benchmark datasets. Comparison for dynamic scene segmentations are shown in Table 4 and Figure 16, 17. Results for multi-view segmentation of static scenes are more accurate than Djelouah, Mustafa, and Guillemaut, and comparable to Kowdle with improved segmentation of some detail such as the back of the chair. We also perform ablative analysis on Equation 2 by removing $E_{data}$, $E_{smooth}$ and $E_{contrast}$ terms. Results demonstrate that joint depth ($E_{data}$, $E_{smooth}$) and segmentation estimation improves the result and contrast information ($E_{contrast}$) helps in improving the quality of the segmentation.

For dynamic scenes the geodesic star convexity based optimization together with temporal consistency gives improved segmentation of fine detail such as the legs of the table in the Office dataset and limbs of the person in the Juggler, Magician and Dance2 datasets in Figure 16 and 17. This overcomes limitations of previous multi-view per-frame segmentation.

4.2 Reconstruction evaluation

Reconstruction results obtained using the proposed method are compared against Mustafa MustafaICCV15 , Guillemaut Guillemaut3dv , and Furukawa Furukawa2010 for dynamic sequences. Furukawa Furukawa2010 is a per-frame multi-view wide-baseline stereo approach which ranks highly on the middlebury benchmark Seitz06 but does not refine the segmentation.

The depth maps obtained using the proposed approach are compared against Mustafa and Guillemaut in Figure 18. The depth map obtained using the proposed approach are smoother with low reconstruction noise compared to the state-of-the-art methods. Figure 19 and 20 present qualitative and quantitative comparison of our method with the state-of-the-art approaches.

Comparison of reconstructions demonstrates that the proposed method gives consistently more complete and accurate models. The colour maps highlight the quantitative differences in reconstruction. As far as we are aware no ground-truth data exist for dynamic scene reconstruction from real multi-view video. In Figure 20 we present a comparison with the reference mesh available with the Dance2 dataset reconstructed using a visual-hull approach. This comparison demonstrates improved reconstruction of fine details with the proposed technique.

In contrast to all previous approaches the proposed method gives temporally coherent 4D model reconstructions with dense surface correspondence over time. The introduction of temporal coherence constrains the reconstruction in regions which are ambiguous on a particular frame such as the right leg of the juggler in Figure 19 resulting in more complete shape. Figure 21 shows three complete scene reconstructions with 4D models of multiple objects. The Juggler and Magician sequences are reconstructed from moving handheld cameras.

Computational Complexity: Computation times for the proposed approach vs other methods are presented in Table 5. The proposed approach to reconstruct temporally coherent 4D models is comparable in computation time to per-frame multiple view reconstruction and gives a $\sim$50% reduction in computation cost compared to previous joint segmentation and reconstruction approaches using a known background. This efficiency is achieved through improved per-frame initialisation based on temporal propagation and the introduction of the geodesic star constraint in joint optimisation. Further results can be found in the supplementary material.

Temporal coherence: A frame-to-frame alignment is obtained using the proposed approach as shown in Figure 22 for Dance1 and Juggle dataset. The meshes of the dynamic object in Frame 1 and Frame 9 are color coded in both the datasets and the color is propagated to the next frame using the dense temporal coherence information. The color in different parts of the object is retained to the next frame as seen from the figure. The proposed approach obtains sequential temporal alignment which drifts with large movement in the object, hence successive frames are shown in the figure.

Limitations: As with previous dynamic scene reconstruction methods the proposed approach has a number of limitations: persistent ambiguities in appearance between objects will degrade the improvement achieved with temporal coherence; scenes with a large number of inter-occluding dynamic objects will degrade performance; the approach requires sufficient wide-baseline views to cover the scene. Background reconstruction is limited to coarse reconstruction of orthogonal planes based on the Manhattan world assumption.

5 Applications

The 4D meshes generated from the proposed approach can be used for applications in immersive content production such as FVV rendering and VR. Unlike the previous methods proposed framework does not require strong prior assumptions and manual interactions to create 4D meshes for real-world applications. This section demonstrates the results of these applications.

5.1 Free-viewpoint rendering

In FVV, the virtual viewpoint is controlled interactively by the user. The appearance of the reconstruction is sampled and interpolated directly from the captured camera images using cameras located close to the virtual viewpoint FVV .

The proposed joint segmentation and reconstruction framework generates per-view silhouettes and a temporally coherent 4D reconstruction at each time instant of the input video sequence. This representation of the dynamic sequence is used for FVV rendering. To create FVV, a view-dependent surface texture is computed based on the user selected virtual view. This virtual view is obtained by combining the information from camera views in close proximity to the virtual viewpoint FVV . FVV rendering gives user the freedom to interactively choose a novel viewpoint in space to observe the dynamic scene and reproduces fine scale temporal surface details, such as the movement of hair and clothing wrinkles, that may not be modelled geometrically. An example of a reconstructed scene and the camera configuration is shown in Figure 23.

A qualitative evaluation of images synthesised using FVV is shown in Figure 24 and 25. These demonstrate reconstruction results rendered from novel viewpoints from the proposed method against Mustafa Mustafa16 and Guillemaut Guillemaut2010 on publicly available datasets. This is particularly important for wide-baseline camera configurations where this technique can be used to synthesize intermediate viewpoints where it may not be practical or economical to physically locate real cameras.

5.2 Virtual reality rendering

There is a growing demand for photo-realistic content in the creation of immersive VR experiences. The 4D temporally coherent reconstructions of the dynamic scenes obtained using the proposed approach enables the creation of photo-realistic digital assets that can be incorporated into VR environments using game engines such as Unity and Unreal Engine, as shown in Figure 26 for single frame of four datasets and for a series of frames for Dance1 dataset.

In order to efficiently render the reconstructions in a game engine for applications in VR, a UV texture atlas is extracted using the 4D meshes from the proposed approach as a geometric proxy. The UV texture atlas at each frame are applied to the models at render time in unity for viewing in a VR headset. A UV texture atlas is constructed by projectively texturing and blending multiple view frames onto a 2D unwrapped UV texture atlas, see Figure 27. This is performed once for each static object and at each time instance for dynamic objects allowing efficient storage and real-time playback of static and dynamic textured reconstructions within a VR headset.

6 Conclusion

This paper introduced a novel technique to automatically segment and reconstruct dynamic scenes captured from multiple moving cameras in general dynamic uncontrolled environments without any prior on background appearance or structure. The proposed automatic initialization was used to identify and initialize the segmentation and reconstruction of multiple objects. A framework for temporally coherent 4D model reconstruction of dynamic scenes from a set of wide-baseline moving cameras. The approach gives a complete model of all static and dynamic non-rigid objects in the scene. Temporal coherence for dynamic objects addresses limitations of previous per-frame reconstruction giving improved reconstruction and segmentation together with dense temporal surface correspondence for dynamic objects. A sparse-to-dense approach is introduced to establish temporal correspondence for non-rigid objects using robust sparse feature matching to initialise dense optical flow providing an initial segmentation and reconstruction. Joint refinement of object reconstruction and segmentation is then performed using a multiple view optimisation with a novel geodesic star convexity constraint that gives improved shape estimation and is computationally efficient. Comparison against state-of-the-art techniques for multiple view segmentation and reconstruction demonstrates significant improvement in performance for complex scenes. The approach enables reconstruction of 4D models for complex scenes which has not been demonstrated previously.

Acknowledgements.

This research was supported by the Royal Academy of Engineering Research Fellowship RF-201718-17177 and the EPSRC Platform Grant on Audio-Visual Media Research EP/P022529.