Warp and Learn: Novel Views Generation for Vehicles and Other Objects

TL;DR

This paper presents a semi-parametric, self-supervised method for generating novel views of vehicles and objects from a single image by integrating geometric priors with deep learning, enabling realistic and versatile viewpoint synthesis.

Contribution

It introduces a semi-parametric approach combining geometric knowledge with deep learning for novel view synthesis, handling arbitrary rotations and shape transfer.

Findings

Outperforms state-of-the-art methods quantitatively.

Preserves high-frequency textures and details.

Extends to various rigid objects with different topologies.

Abstract

In this work we introduce a new self-supervised, semi-parametric approach for synthesizing novel views of a vehicle starting from a single monocular image. Differently from parametric (i.e. entirely learning-based) methods, we show how a-priori geometric knowledge about the object and the 3D world can be successfully integrated into a deep learning based image generation framework. As this geometric component is not learnt, we call our approach semi-parametric. In particular, we exploit man-made object symmetry and piece-wise planarity to integrate rich a-priori visual information into the novel viewpoint synthesis process. An Image Completion Network (ICN) is then trained to generate a realistic image starting from this geometric guidance. This careful blend between parametric and non-parametric components allows us to i) operate in a real-world scenario, ii) preserve high-frequency visual information such as textures, iii) handle truly arbitrary 3D roto-translations of the input and iv) perform shape transfer to completely different 3D models. Eventually, we show that our approach can be easily complemented with synthetic data and extended to other rigid objects with completely different topology, even in presence of concave structures and holes (e.g. chairs). A comprehensive experimental analysis against state-of-the-art competitors shows the efficacy of our method both from a quantitative and a perceptive point of view. Supplementary material, animated results, code and data are available at: https://github.com/ndrplz/semiparametric

Tables

Table 1. TABLE I: Fréchet Inception Distances  [ 20 ] results for car . Each row lists the average distance between real and generated images for each method on the left. Results are reported from 12 evenly spaced azimuthal angles while rotating around the object at fixed elevation and radius. Details in Sec.  4.2 .

	0°	30°	60°	90°	120°	150°	180°	210°	240°	270°	300°	330°	Avg
oursreal+synth	174.2	57.0	47.9	53.8	52.9	59.9	168.8	61.2	52.1	54.2	46.8	60.2	74.1
oursreal	178.9	54.6	51.0	53.2	56.8	58.1	202.5	62.2	54.1	49.8	46.3	56.1	77.0
MV2NV [57]	144.5	170.9	136.5	192.3	144.7	163.1	152.2	179.1	137.5	186.0	136.0	161.3	158.7
MV3D [59]	257.5	263.9	259.7	284.2	273.0	271.9	267.7	263.7	261.5	277.9	266.5	265.3	267.7
TVSN [43]	70.8	71.3	74.1	78.9	79.0	82.2	89.4	81.1	79.0	78.9	78.0	72.9	78.0
VUnet [11]	202.4	90.7	79.9	88.3	78.8	96.3	203.3	94.0	77.5	85.2	82.0	92.6	105.9
VON [80]	134.1	107.2	150.0	126.5	124.4	114.4	151.7	113.8	127.2	128.9	132.0	107.3	126.4
VONFT [80]	165.2	100.6	137.8	125.9	137.6	108.1	190.5	155.7	134.8	123.1	117.1	102.1	133.2

Table 2. TABLE II: Blind randomized A/B test results. Each row lists the percentage of workers who preferred the novel viewpoint generated with our method with respect to each baseline (chance is at 50%).

	Car (plain)	Car (textured)	Avg
ours > VUnet [11]	76.0%	85.0%	78.0%
ours > VON [80]	88.0%	98.0%	91.0%
ours > VONFT [80]	96.0%	99.0%	97.0%

Table 3. TABLE III: Fréchet Inception Distances  [ 20 ] results for chair class. Each row lists the average distance between real and generated images for each method on the left. Results are reported from 12 evenly spaced azimuthal angles while rotating around the object at fixed elevation and radius.

	0°	30°	60°	90°	120°	150°	180°	210°	240°	270°	300°	330°	Avg
oursreal+synth	83.9	57.9	66.5	120.1	82.5	76.1	102.3	77.2	83.4	117.0	65.2	60.9	82.8
oursreal	92.6	62.4	68.0	122.5	84.8	84.7	117.3	82.3	85.0	127.5	70.1	64.0	88.4
TVSN [43]	84.7	86.6	90.8	95.1	94.0	97.2	93.9	94.6	95.0	93.0	87.8	82.7	91.6
VON [80]	125.5	107.7	100.1	121.1	122.6	136.8	203.9	141.4	123.8	114.2	102.1	96.1	124.6
VUnet [11]	118.9	97.1	123.1	171.0	160.5	137.9	151.2	141.9	155.8	154.9	120.1	95.8	135.7
MV2NV [57]	180.3	168.0	178.3	187.8	184.1	177.8	184.7	184.3	200.2	191.0	184.7	166.4	182.3

Equations

$${\mathcal{P}}_s=\{{\mathbf{p}}_s^{(0)},{\mathbf{p}}_s^{(1)},\dots ,{\mathbf{p}}_s^{(|\mathcal{P}|)}\},{\mathbf{p}}_s^{(i)}=\Gamma ({\mathbf{k}}_s^{(i)})$$

$${\mathbf{V}}_d=\left[\begin{array}{cc}\mathbf{R}& \mathbf{t}\\ 0^T& 1\end{array}\right]$$

$$k_d^{(i)}=\left[\begin{array}{ccc}f& 0& c_x\\ 0& f& c_y\\ 0& 0& 1\end{array}\right]{\mathbf{V}}_d^{-1}{k}_{3D}^{(i)}$$

$${\mathcal{P}}_d=\{{\mathbf{p}}_d^{(0)},{\mathbf{p}}_d^{(1)},\dots ,{\mathbf{p}}_d^{(|\mathbf{P}|)}\},{\mathbf{p}}_d^{(i)}={\mathbf{H}}^{(i)}({\mathbf{p}}_s^{(i)})$$

$${\tilde{\mathcal{P}}}_{\mathbf{s}}=\{{\tilde{\mathbf{p}}}_s^{(0)},{\tilde{\mathbf{p}}}_s^{(1)},\dots ,{\tilde{\mathbf{p}}}_s^{(|\mathbf{P}|)}\},{\tilde{\mathbf{p}}}_s^{(i)}=({\mathbf{H}}^{(i)}{)}^{-1}({\mathbf{p}}_d^{(i)})$$

$$\mathcal{C}=\{C^{(0)},C^{(1)},\dots ,C^{(|\mathcal{C}|)}\},C^{(i)}\in {\mathrm{I}\mathrm{R}}^{faces\times 3\times 3}$$

$$s_{2.5D}(C^{(\alpha )},{\mathbf{V}}^{(i)})\in {\mathrm{I}\mathrm{R}}^{H\times W\times 3}$$

$${\tilde{\mathbf{x}}}_s=g({\tilde{\mathcal{P}}}_{\mathbf{s}},s_{2.5D}(C^{(\alpha )},{\mathbf{V}}_s),c_s|\theta )$$

$${\mathcal{L}}_{{\mathbf{x}}_s,{\tilde{\mathbf{x}}}_s}^{VGG}(\theta )=\sum _l{\lambda }_l\Vert {\Phi }_l({\mathbf{x}}_s)-{\Phi }_l({\tilde{\mathbf{x}}}_s){\Vert }_1.$$

$${\mathcal{L}}_{{\mathbf{x}}_s,\tilde{\mathbf{x}}}^{adv}={\mathbb{E}}_{{\mathbf{x}}_s}[\log D({\mathbf{x}}_s)]+{\mathbb{E}}_{\tilde{\mathbf{x}}}[\log (1-D(\tilde{\mathbf{x}}))]$$

$$\mathcal{L}={\mathcal{L}}_{{\mathbf{x}}_s,{\tilde{\mathbf{x}}}_s}^{VGG}+\gamma {\mathcal{L}}_{{\mathbf{x}}_s,\tilde{\mathbf{x}}}^{adv}$$

$$\mathrm{FID}=\|\bm{m}-\bm{m}_{w}\|_{2}^{2}+\mathrm{Tr}\bigl{(}\bm{C}+\bm{C}_{w}-2\bigl{(}\bm{C}\bm{C}_{w}\bigr{)}^{1/2}\bigr{)}$$

Full text

Supplementary material, animated results, code and data are available at:

https://github.com/ndrplz/semiparametric

1 Introduction

How would you see an object from another point of view? Given a single view of an object in the world, predicting how it would look like from arbitrarily different viewpoints is definitely non-trivial for both humans and machines. Still, people with a good visual-spatial intelligence [14] can easily imagine objects’ rotation, zoom and translation shifts, especially if objects have a well known shape and feature some degree of symmetry. Indeed, humans have been shown to perform mental transformations for decision-taking about their surrounding environment [51, 3].

Machines are still far from this level of intelligence.

Still, powerful parametric (i.e. entirely learning-based) deep learning models [28, 16] made it possible to frame the generation of novel viewpoints as a conditioned image synthesis problem. However, this is an holistic approach that under-exploits the fact that man-made objects 3D models are roughly distributed according to few prototypes (e.g. sedan, VAN, pick-up, truck etc. for vehicles). Up to now, the generation is constrained to be visually plausible with almost no geometric support from prototypes’ shapes [59, 43]. Furthermore, even though generated images may look realistic per se, fine-grained visual appearance characterizing the particular object instance (e.g. texture) is often lost due to its high frequency which hardly survives being encoded through a deep network [59, 80, 11]. Eventually, most methods are supervised on the target image, which is seldom available for real data. Also, vast amount of data are required for the network to generalize to arbitrary transformations (i.e. a sufficient number of images for every possible viewpoint). This constrains many methods to be trained solely on synthetic data and to be restricted to a discrete set of viewpoints: in fact, most recent works only handle a small set of transformations (e.g. fixed radius rotations) [70, 59, 77, 43].

At the same time, an independent line of research has shown that a non-parametric approach can be a viable path for photorealism, as also pointed out by Qi et al. [46]. For instance, new images can be generated by collaging [17, 29, 7, 22] or by leveraging multiple photographs to synthesize novel views via image-based rendering [5, 40, 19, 41]. Still, these methods require a large amount of data at test time: entire image banks for collaging, multiple photographs and depth data for image-based rendering.

In this work we propose a new approach - inherently semi-parametric - being based on both learning and geometry, self-supervised and efficient to be used in real-time. By taking the best from both worlds, we exploit geometric constraints to roughly sketch the target shape of the object and its textures while still relying on deep view synthesis to refine the generated view. The rationale behind this work is that many man-made objects adhere to a-priori geometric rules: vehicles in particular, exhibit a symmetric, piece-wise planar structure. Therefore, those properties may be exploited to approximately represent them by a small set of piece-wise planar patches, which can be warped almost exactly from source to destination viewpoint via a symmetry-aware homography transformation. Although these warped patches provide a rich hint about the visual content of the target viewpoint, they are far from being useful on their own. Thus, a fully-convolutional network is seeded with these patches along with a 2.5D CAD-rendered sketch to be used as guidance; it is then trained in a self-supervised manner to discriminate which part of the image must be completed or in-painted for the result to look realistic (see Fig. 3).

We tested our solution in particular in vehicle generation due to their ubiquity in urban scene understanding applications [71, 60, 13, 50, 35, 31]. Moreover, to highlight that a decomposition in planar patches holds for different types of rigid objects, we evaluate our semi-parametric framework on both convex objects (vehicles) and concave ones (chairs). We leave as future work the analysis of a broader set of object categories.

In summary, our main contributions follow:

• •

We propose an original formulation of the problem of object novel viewpoint synthesis in a semi-parametric setting. Loose geometrical assumptions about the object shape provide rich hints about its appearance (non-parametric); this information guides a fully-convolutional network (parametric) in the generation process.

• •

We design our model to be trainable on existing datasets for 3D object detection in a self-supervised manner, with no need for paired source/target viewpoint images. Furthermore, we leverage 2D keypoints for real-world images where foreground segmentation is not provided.

• •

We demonstrate how our method excels in preserving visual details (e.g. texture) and in performing realistic shape transfer to completely different 3D models, while still being resilient to a much wider range of 3D transformations than competitors.

Our method can be employed to generate realistic novel views of an object from an arbitrary zoom, viewpoint and distance, as depicted in Figures 1, 4, 6, 7, 9, 11, 12. This could enable the generation of novel views from arbitrary virtual cameras in an urban scene, with applications in surveillance, vehicle re-identification and forensics. Also, our approach allows a disentangled editing of object shape and appearance (i.e. shape can be changed while preserving appearance or the other way around). This enables applications in interactive 3D manipulation and design, as well as data augmentation (Fig. 13).

A thorough experimental analysis is conducted comparing our proposal with state-of-the-art methods, considering both the quantitative and the perceptual point of view.

2 Related Work

View synthesis In just few years, the widespread adoption of deep generative models [28, 16] has led to astounding results in different areas of image synthesis [47, 1, 73, 25, 62, 72]. In this scenario, conditional GANs [36] have been demonstrated to be a powerful tool to tackle image-to-image translation problems [23, 78, 79, 9]. Hallucinating novel views of the subject of a photo can be naturally framed as an image-to-image translation problem. For human subjects, this has been cast to predicting the person’s appearance in different poses [34, 53, 74, 11]. Fashion and surveillance domains drew most of the attention, with much progress enabled by large real-world datasets providing multiple views of the same subject [32, 75].

For rigid objects instead, this task is usually referred to as novel 3D view synthesis and additional assumptions such as object symmetry are taken into account. In point of fact, symmetry is the most common assumption [77, 43, 80] to synthesise disoccluded portions of the object. Starting from a single image, Yang et al. [70] showed how a recurrent convolutional network can be trained via curriculum-learning to perform out-of-plane object rotation. In a similar setting Tatarchenko et al. [59] predicted both object appearance and depth map from different viewpoints. Successive works [77, 43] trained a network to learn a symmetry-aware appearance flow, re-casting the remaining synthesis as a task of image completion; [57] extends this approach to the case in which $N>1$ input viewpoints are available. However, all these works [70, 59, 77, 43, 57] assume the target view to be known at training time. As this is not usually the case in the real-world, these approaches limit themselves to train solely on synthetic data and exhibit limited generalization in a real-world scenario. The recent work by Zhu et al. [80] exploits cycle consistency losses to overcome the need of paired data, thus training on datasets of segmented real-world cars and chairs they gathered for the purpose. Although that work shows more realistic results, it requires pixel-level segmentation for each class of interest. In contrast, we show that already available datasets for object 3D pose estimation [68, 67] can be used for this purpose, despite the extremely rough alignment between the annotated model and the image.

3D Shape reconstruction Few recent works [64, 80] have shown that the use of 2.5D sketches can be a viable path to bridge the gap between synthetic and real-world data. In particular, in Zhu et al. [80] the 2.5D sketch consists of both a silhouette and a depth image rendered from a learnt low-resolution voxel grid by means of a differentiable ray-tracer. While this method is appealing for its geometrical guarantees, it is limited by a number of factors: i) it requires a custom differentiable ray-tracing module; ii) footprint of voxel-based representations scales with the cube of the resolution despite most of the information lying on the surface [55, 42]; iii) errors in the 3D voxel grid naturally propagate to the 2.5D sketch. We also follow this line of work to provide soft 3D priors to the synthesis process. However, in our semi-parametric setting 2.5D sketches are additional inputs which can be rendered from arbitrary viewpoints using standard rendering engines.

Nonparametric view synthesis In the interactive editing setting, recent works [26, 48] have shown astounding results by keeping the human in the loop and assuming a perfect (even part-level) alignment between the 3D model and the input image. However, as pixels are warped from the input to the target view [48] it is not feasible to perform shape transfer to a completely different model. Moreover, the time required to synthesise the output is still far from real-time (few seconds). On the opposite, our method enables disentangled shape and appearance transfer in real-time, with only a coarse alignment between the input image and the 3D model.

In the scenario of image synthesis from semantic layout the recent work of Qi et al. [46] has shown that non-parametric components (i.e. a memory bank of image segments) can be integrated in a parametric image synthesis pipeline to produce impressive photo-realistic results. Despite our different setting, we similarly rely on image patches to provide hints to the Image Completion Network (ICN); however, our patches are not queried from a database but warped directly from the input view.

After our initial submission, several new related works have been published by the scientific community. We include here a non exhaustive list of them; [37, 69, 56, 38, 39, 8].

3 Model

Our model generates novel views of objects in a semi-parametric setting - relying on both geometry and learning. To this end, 2D keypoints and additional 3D information are extracted from a single view of the object. Keypoints are used as a proxy to describe 2D geometrical abstractions of the 3D shape (i.e. planar patches), which are transformed to the novel viewpoint. Eventually, a convolutional neural network seamlessly fuses this prior information to generate a realistic image from the novel view.

More into details, our model takes as input an image depicting a single object $\mathbf{x}_{s}\in{\rm I\!R}^{H\!\times W\!\times 3}$ viewed from the source viewpoint $\mathbf{V}_{s}\in{\rm I\!R}^{4\!\times 4}$, its 2D keypoints $\bm{\mathcal{K}}_{s}$ and its associated 3D CAD model $C\in{\rm I\!R}^{faces\times 3\times 3}$ having 3D keypoints $\bm{\mathcal{K}}_{3D}$.

Training (Fig. 3, top) is performed in a self-supervised fashion maximizing the consistency between the input image and the generated one when projected onto the source viewpoint, with no need for coupled images from the two viewpoints as supervision. Given $\mathbf{x}_{s}$ and $\bm{\mathcal{K}}_{s}$, planar patches are extracted (Sec. 3.1). The patches are then projected to the target viewpoint $\mathbf{V}_{d}$ through an intermediate view (Sec. 3.2) according to a visibility model (Sec. 3.3). The 3D model $C$ is also rendered from the target viewpoint $\mathbf{V}_{d}$ to get a 2.5D sketch of the object (Sec. 3.4). Eventually, the image completion network (ICN) starts from these visual seeds to generate a realistic final image (Sec. 3.5, 3.6).

Inference (Fig. 3, bottom) follows a similar flow. However, in this case only 2D keypoints $\bm{\mathcal{K}}_{s}$ and source viewpoint $\mathbf{V}_{s}$ are needed; the 3D model can be either inferred from the input or arbitrarily selected to perform shape transfer.

Without loss of generality, in this work we rely on ground truth data whenever possible, as our focus lies on the overall viewpoint generation pipeline. Off-the-shelf detectors [18, 45, 61] can be used to provide these information in an in-the-wild scenario.

3.1 Keypoint-based decomposition into planar patches

We leverage 2D keypoints to approximate the visible shape of the object with a simple polyhedron with a small set of faces, as depicted in Fig. 2. Since keypoints mark characteristic locations in the object shape (e.g. corners), a face defined from at least three of those could carry a perceptual / semantic meaning (e.g. the roof of a car). Exploiting 2D keypoints to find object faces is appealing for a number of reasons. First, this makes straightforward computing the homography matrix between planes in different viewpoints (see Sec. 3.2). Furthermore, a number of datasets provide object landmark annotations in real-world scenarios (e.g. [30, 68, 67, 63, 65, 21]) and solid keypoints detection methods exist [61, 45, 18].

Specifically, for each source image $\mathbf{x}_{s}\in{\rm I\!R}^{H\!\times W\!\times 3}$ an array of 2D keypoints $\bm{\mathcal{K}}_{s}\in{\rm I\!R}^{|\bm{\mathcal{K}}_{s}|\!\times 2}$ is available, being $|\bm{\mathcal{K}}_{s}|$ the category-specific number of keypoints (we set $|\bm{\mathcal{K}}_{s}|=12$ for vehicles). From these a set of planar patches are extracted according to the geometry of the object:

[TABLE]

where each patch $\mathbf{p}_{s}^{(i)}$ is defined as the convex hull $\Gamma$ of the subset of keypoints $\bm{k}_{s}^{(i)}\subseteq\bm{\mathcal{K}}_{s}$. Prior knowledge about the object class can be leveraged to choose the planar patches $\bm{\mathcal{P}}_{s}$: for instance, we choose roof, left, right, front and back sides for vehicles.

3.2 Warping and dewarping

Warping patches Source patches $\bm{\mathcal{P}}_{s}$ are warped to the destination viewpoint to get a set of warped patches $\bm{\mathcal{P}}_{d}$ that are employed to bootstrap the novel viewpoint synthesis. To this end, we define the destination viewpoint $\mathbf{V}_{d}\in{\rm I\!R}^{4\!\times 4}$ to be an arbitrary rigid transformation of the camera:

[TABLE]

Locations of 2D keypoints $\bm{\mathcal{K}}_{d}\in{\rm I\!R}^{|\bm{\mathcal{K}}_{s}|\!\times 2}$ in the novel viewpoint can be now computed by using the classical pinhole camera model as:

[TABLE]

where $k_{3D}^{(i)}$ is the $i^{th}$ 3D keypoint in the CAD model and $c_{x}$, $c_{y}$ are the principal point coordinates. As the focal $f$ is unknown, we set it to an high value to minimize perspective effects; we choose $f=3000$ as in Pascal3D+ [68]. A set of homography transformations $\mathbf{H}$ relating planar surfaces in the two views can be estimated from correspondences between $\bm{\mathcal{K}}_{s}$ and $\bm{\mathcal{K}}_{d}$. In this way patches in the destination viewpoint (warped patches from now on) can be computed as:

[TABLE]

De-warping patches Since real-world datasets do not provide paired views, it is not possible to supervise the destination image $\tilde{\mathbf{x}}_{d}$; hence, we propose to train the ICN network in a self-supervised manner. A straightforward approach would be to reconstruct $\mathbf{x}_{s}$ from $\bm{\mathcal{P}}_{s}$. Nonetheless, this would create a distribution shift between the data fed to the network during training and inference stages. In fact, while $\bm{\mathcal{P}}_{s}$ is perfectly aligned with $\mathbf{x}_{s}$, $\bm{\mathcal{P}}_{d}$ might be affected by homography failures and interpolation errors among other issues. For example, when the destination area is smaller than the source one many source pixel land onto the same destination pixel, and the inverse warping cannot recover all the information in the original patch. To alleviate this shift, we train the network to reconstruct the image $\mathbf{x}_{s}$ from a third set of patches (called dewarped patches in what follows):

[TABLE]

During training, patches are warped towards a random viewpoint sampled from the training set distribution before being warped back to the source one. In this way the network learns to cope with possible transformation errors and cannot simply short-circuit input patches to the output. The importance of this dewarping trick for a well-behaved network training is highlighted in Sec. 4.1.

3.3 Visibility model

Whenever a 3D object is projected into a 2D image, self-occlusions almost inevitably arise. Consequently, not all planar patches into the set $\bm{\mathcal{P}}_{s}$ are effectively visible. Were they to be warped regardless of their visibility, following parts of our architecture would require to discern which of them to keep or discard. Furthermore, when warping between $\bm{\mathcal{P}}_{s}$ and $\bm{\mathcal{P}}_{d}$ the visibility of some of those planes may vary. To take these dynamics into account, we first render the object 3D model from the camera viewpoint to obtain the 3D planes corresponding to the detected 2D patches. The z-buffer computed through ray-casting is then exploited to filter the patches which are not visible from the source viewpoint. These are “dropped”, in the sense that they are zeroed before feeding them to the ICN. As during training the intermediate viewpoint $\mathbf{V}_{d}$ is sampled randomly, the warping-dewarping phase results in a random dropout at patch-level, where the chance of drop is inversely proportional to the frequency of visibility of the patch. This forces the network to hallucinate missing patches during training, thus improving generalization when source and destination viewpoint differ.

3.4 Leveraging 2.5D sketches

While image patches carry rich information about the appearance of the object, they bear few cues about the object shape. In other words, visual aspect and shape are disentangled by design. This is a desirable property enabling multiple applications which require to change one of the two while keeping the other fixed. In this section we propose a method to constrain the synthesised object shape. Let

[TABLE]

be the set of 3D CAD models which approximate the intra-class variation for the current object class, each $C^{(i)}$ being a 3D mesh composed of $f$ faces. The number of CADs $|\mathcal{C}|$ needed to cover the intra-class variation reasonably depends on the object category, but it is often relatively low (e.g. $|\mathcal{C}|=10$ for the vehicle class in the Pascal3D+ dataset [68]). Each training example $i$ is thus composed by an image $\mathbf{x}^{(i)}$ and its associated viewpoint $\mathbf{V}^{(i)}$ and CAD index $\alpha\in\{0,1,\dots,|\mathcal{C}|\}$ which can be possibly selected through a classifier. Therefore, a virtual camera can be used to render the CAD $C^{(\alpha)}$ from viewpoint $\mathbf{V}^{(i)}$. In particular, following [64], we render the 2.5D sketch of CAD surface normals:

[TABLE]

which provides rich information about the object’s 3D shape. During training, this 2.5D sketch is fed to the ICN together with de-warped patches $\bm{\tilde{\mathcal{P}}_{s}}$ to reconstruct $\mathbf{x}_{s}$.

3.5 Appearance prior

Our method relies on warped patches to transfer the object appearance from a source to a destination viewpoint. Still it might happen that viewpoints $\mathbf{V}_{s}$ and $\mathbf{V}_{d}$ are so far apart (e.g. front to back) that an object shares no visible faces across the two even with symmetry constraints. To alleviate this issue, we crop from the center of the input image $\mathbf{x}_{s}$ a small patch $c_{s}$ with side $10\%$ of the image size and give it as an additional input to the image completion network as a prior knowledge about the rough object appearance in absence of other hints (depicted in Fig. 2). The network can extract from this crop coarse information about the object visual aspect (e.g. the average color) as a prior to cope with large changes between viewpoints.

3.6 Image Completion Network

The Image Completion Network (ICN) $g(\cdot\,\,|\,\,\theta)$ is a fully convolutional network parametrized by $\theta$ trained to reconstruct a realistic image $\mathbf{x}_{s}$ from dewarped patches $\bm{\tilde{\mathcal{P}}_{s}}$, 2.5D sketch $s_{2.5D}$ and appearance prior $c_{s}$:

[TABLE]

Architecture Our ICN features an encoder / decoder structure as in [78, 80]. The encoder is composed of 3 convolutional blocks to reduce the spatial resolution and 3 additional residual blocks applied to the lowest resolution code. Except for the first one, every convolution has kernel size 4x4 and it’s preceded by reflection padding, while ReLU activation and Instance normalization are applied afterwards. The decoder follows the same structure in reverse order. For the discriminator network we rely on the two-scale PatchGAN classifier [23, 78, 80].

Objective A number of recent works [24, 6, 46] indicate that loss functions based on high-level features extracted from pretrained networks can lead to much more realistic results compared to naive per-pixel losses between the output and ground-truth image. Given a set of layers $\{\Phi_{l}\}$ from a network $\Phi$ and a training pair consisting of a real and a generated images $(\mathbf{x}_{s},\tilde{\mathbf{x}}_{s})$, we define the perceptual loss function as

[TABLE]

We employ each second convolutional layer of each block in VGG-19 [54] as feature extractor $\Phi_{l}$; $\{\lambda_{l}\}$ is set as in [6].

As mentioned above, images generated from novel viewpoints $\tilde{\mathbf{x}}_{d}$ cannot be directly supervised if the dataset does not provide paired views. Nevertheless, we can still enforce the realism of ICN output in an adversarial fashion. Given a generic image $\tilde{\mathbf{x}}$ synthesised by ICN either in the source ($\tilde{\mathbf{x}}_{s}$) or the destination ($\tilde{\mathbf{x}}_{d}$) viewpoint, we set up a min-max game as follows:

[TABLE]

where D is the discriminator network aiming to distinguish between real and synthesised images. Our total loss is defined as:

[TABLE]

where $\gamma$ modulates the contribution of the adversarial term.

4 Experiments

The experiments we propose concern with the evaluation of the quality of the generated object images across different viewpoints. First, both visual comparison (Sec. 4.1) and quantitative experiments (Sec. 4.2) against state-of-the-art competitors are reported. Then, we keep the human in the loop by relying on human judgement to measure the output quality via A/B preference tests (Sec. 4.3). Eventually, we extend the evaluation to other classes (Sec. 4.4) and we investigate the contribution of complementary synthetic data for modelling extreme viewpoint changes (Sec. 4.5).

Datasets Although large-scale 3D shape repositories providing object geometries such as Princeton Shape Benchmark [52] and Shapenet [4] exist, they do not come with real-world images aligned. As we want to work with real-world data, we rely on Pascal3D+ [68], an in-the-wild 3D object detection dataset which augments the 12 rigid categories of the PASCAL Visual Object Classes (VOC) [12] with 3D annotation -roughly [58]- aligned. In particular, we use the car and chair subsets, which consist of around 5000 and 1500 images respectively.

Competitors We evaluate our method against six state-of-the-art works in the task of novel viewpoint synthesis. The first one is VON [80], an adversarial learning framework in which object shape, viewpoint and texture are treated as three conditionally independent factors that contribute to the synthesis of the novel viewpoint. Since VON was originally trained on a custom car dataset collected by the authors [80], for a fair comparison we implement a second baseline VONFT by fine-tuning their network on Pascal3D+. For both competitors we provide ground truth voxelized shapes from  [80] matching the images’ CADs, relying on the texture encoder network from [80] for extracting the textures from Pascal3D+ images. Third, we compare to VUnet [11], a state-of-the-art framework for conditional image generation based on variational autoencoder [28], which shows a good generalization capability across a variety of poses and viewpoints. In the authors’ implementation [11] a U-net [49] architecture is fed with keypoint-based skeletons to perform pose-guided human generation. We re-train their model on Pascal3D+ to perform pose-guided object generation. For this process, we feed their shape-encoding network with our 2.5D sketches rendered from the object CAD - which is a more informative signal than the one used in the original implementation (i.e. skeleton or edges) as it also includes the direction of the normal surfaces and fine-grained details from the 3D model.

We also compare with three recent pairwise-trained models: MV3D [59], TVSN [43] and MV2NV [57]. Pairwise methods share the need for both source and target pairs during training; thus we cannot re-train or finetune them on Pascal3D+ and we rely on pre-trained models released by the authors.

To maximize evaluation fairness, in what follows we only sample novel viewpoints rotating around the z-axis at fixed distance and elevation, which is the only setting handled by competitors. Still, our method can handle general roto-translations as well as variation in camera intrinsic. Fig. 7 and Fig. 9 show visual results for large viewpoint changes in both elevation and azimuth; even more extreme transformations are depicted in Fig. 12.

Implementation details The 2D bounding box of each example of Pascal3D+ [68] is padded to a squared aspect ratio and resized to 128x128 pixels. We work in LAB space relying on the training procedure from [66]. Following [61, 45] truncated and occluded objects are discarded, resulting in 4081 training and 1042 testing examples respectively. For vehicles, the oursreal+synth model is trained for 20 epochs with batch size 8. During training images undergo small random rotations, translations and shearing for data augmentation purposes. We use Adam [27] optimizer with constant learning rate $2e^{-4}$. Loss balancing term $\gamma$ is set to $8$. The code is developed in PyTorch [44]: we depend on Open3D library [76] for 3D data manipulation and rendering. Random search has been employed for hyper-parameter tuning. Without any optimization, inference for a 128x128 image takes $\sim 3ms$ on a NVIDIA GTX 1080, making it suitable for real-time applications.

4.1 Visual results

Our model produces high-quality results for a variety of camera viewpoints, preserving fine-grained object appearance, as it can be appreciated in Figures 1, 4, 5, 6, 7, 11 and 12.

Competitors The key differences between the proposed method and competitors can be appreciated in Fig. 6. From left to right. MV2NV [57] seems to suffer the most the reality gap as well as the lack of multiple views, leading to the worse quality results in our setting. MV3D [59] predicts at least a reasonable overall shape, although images are blurry to the point that in some cases one can barely recognize the vehicle. Results from TVSN [43] show the highest variability: while looking generally fine, they are disastrous for less common poses such as (b) and (e). The output from Visual Object Networks [80] (VON) is generally realistic, but hardly reflects the visual appearance of the input. Furthermore, both VON and VONFT generator networks do not generalize to poses which are less common in the training set such as the frontal pose in (d). VUnet [11] suffers from blurred results typical from variational autoencoders [15]; also, due to skip connections, input appearance may leak to the output when the two viewpoints are very different (b). More generally, the drawbacks of a solely learning-based viewpoint synthesis are evident in (a, c): complex textures cannot be recovered once compressed in a feature vector.

Ablation Ablated versions of our model are shown in Fig. 5. The effect of removing the appearance prior is showcased in No-prior column. Without prior information, the ICN fails to infer the object appearance when no planar patch is provided, as shown in (b, e). Removing the adversarial term (No-adv column) results in slightly blurred outputs. We also investigate the aid of the dewarping trick presented in Sec. 3.2. In No-dewarp column, the ICN was trained to reconstruct the image from $\bm{\mathcal{P}}_{s}$ instead of $\bm{\tilde{\mathcal{P}}_{s}}$ (see Sec. 3.1). As expected, despite the very low reconstruction error at training time (due to the similarity between $\mathbf{x}_{s}$ and $\bm{\mathcal{P}}_{s}$), the model fails to generalize to the synthesis of novel viewpoints where the textures $\bm{\mathcal{P}}_{d}$ are the result of an homography transformation. This highlights the importance of the dewarping trick for a well-behaved training. Eventually, Sil-only shows the ablated versions in which the input sketch is constituted only by 2D silhouette. Although results do not differ dramatically, it can be appreciated how the network benefits from additional information to resolve ambiguous situations such as self-occlusions (e) and details such as side windows, lights, wheels.

Shape transfer Visual results for shape transfer are showed in Fig. 1. In this setting, the network is requested to complete the warped faces $\bm{\mathcal{P}}_{d}$ using the 2.5D sketch rendered from a totally different CAD. It can be appreciated how novel viewpoints are still realistic, since the network exploits the 2.5D sketch to complete the warped appearance in a CAD-agnostic manner.

4.2 Metrics evaluation

Fréchet Inception Distance To quantitatively measure the similarity between generated and real images we rely on Fréchet Inception Distance (FID), which was shown to consistently correlate with human judgment [20, 33]. We employ activations from the last convolutional layer of an InceptionV3 model pretrained on ImageNet [10] as features. Assuming a multidimensional Gaussian distribution for these features, we compute the FID as follow:

[TABLE]

Where $\bm{m}$, $\bm{C}$ are the mean and covariance of the features extracted from Pascal3D+ data, while $\bm{m}_{w}$ and $\bm{C}_{w}$ are the corresponding statistics extracted from the generated images.

To enable the comparison with other works, we sample novel viewpoints while rotating around the object at fixed distance and elevation: results are reported in Table I, binned in 12 equidistant azimuthal angles. Fréchet Inception Distance rewards the realism we can get with our semi-parametric approach; our method outperforms all competitors for the vast majority of viewpoints. In particular, our method preserves both low (i.e. shape) and high-frequency (i.e. texture) image statistics which are both contribute to the overall FID score.

4.3 Perceptual experiments

To assess the quality of our results also from a perceptual point of view, randomized A/B preference tests were performed by 43 human workers, following the experimental protocol of previous works [6, 46, 80]. Due to time constraint, for this phase we select only the three most recent competitors, namely VON [80], VONFT [80] and VUnet [11]. As we want to evaluate both the realism and the appearance coherence of our method, we perform two different tests.

View transfer coherence In the first setting, the subject is presented with three images: while the first one comes from Pascal3D+ test set, A and B depict a novel viewpoint of the object generated with two different methods. The human worker is then asked whether rotating the input object would better lead to A or B. Results reported in Tab. II indicate that our method is largely preferred to competitors, likely because of the built-in realism that comes from warping the original image. As a further analysis we split by manual annotation Pascal3D+ images into plain and textured sets, the latter set containing vehicles which feature characteristic textures. Table II highlights that workers expressed almost unanimous preference for our method on the textured set. The fact that human attention was caught by these appearance details highlights the importance of preserving fine-grained details in the synthesized output.

Output realism The second experiment consists of a two-alternative forced choice aimed at evaluating the relative realism of each method. Here the subject is presented with only two images for a determined amount of time. The worker is then asked which of the two appeared more realistic and the experiment is repeated by varying the amount of time. Results depicted in Fig. 8 follow two trends. On the one hand, workers clearly discern VUnet and VONFT images from real ones as more time is available. VUnet is hurt by excessive blur and visual artifacts; VONFT suffers from a severe loss of realism w.r.t. the original VON method, which may be related to the great variety of viewpoints in the Pascal3D+ dataset compared with the one used in Zhu et al. [80]. On the other hand, both VON and our method produce realistic images workers struggle to distinguish from the real ones even in 8000ms.

4.4 Evaluation on other classes

Similarly to [57, 43, 80], we test our method also on the chair subset from Pascal3D+ dataset, consisting of 1195 images annotated with 10 different CAD models to assess the generalisation capability of our method. As for vehicles, also for chairs we define a set of planes from 3D annotated keypoints to approximate the surface of the objects, namely: left, right, seat and back. Even though chairs often feature holes and slits leaking some of the background, we treat them as filled planes. This is due to having access to the foreground segmentation mask through the rendered CAD model, which can be used to mask out background areas. However, the very coarse alignment of the chairs CAD models in Pascal3D+ leads to an additional difficulty when training the ICN.

Table III reports results against competitors in terms of Fréchet Inception Distance score: MV3D [59] is omitted since it only releases pre-trained model for cars. Our method outperforms all other competitors also for this class of objects; visual examples are reported in Fig. 11. While MV2NV [57] and TVSN [43] often struggle to generate the object from the correct viewpoint, VON [80] fails to transfer visual details in the final output. Although the generated image looks realistic, it doesn’t resemble the input one. Contrarily, our method successfully generate realistic views of the input object, even under severe viewpoint transformations 12. Still, when background leaks in the input mask due to the CAD misalignment (first and fourth row) the final output quality decreases.

4.5 On the use of synthetic data

Even though the semi-parametric nature of our proposed method copes with a variety of viewpoint, it still rely on data to learn how to stitch together the warped patches. Therefore, for dramatic changes of viewpoint that are completely uncovered in the dataset, performances may drop significantly. As shown in Fig. 10 (a), Pascal3D+ viewpoints’ distribution for the car set is profoundly skewed towards low elevation values and is polarized with regard to the azimuth to frontal and lateral views, reflecting how the images were acquired. As it is of great interest to produce realistic images from more varied viewpoints (e.g. bird’s eye view), we include synthetic data in the training set for balancing the viewpoints’ distribution. To this end, we sample 59 models from the car synset in the ShapeNet [4] dataset and we annotate them with 3D keypoints to define the planes of interest. We then render 6950 images sampling viewpoints uniformly in a 3D semi-spheres around the origin. As shown in Fig. 10 (b) this viewpoints’ distribution is much more uniformly distributed in terms of azimuth and elevation. We name oursreal+synth the ICN network trained on a mixture of synthetic and real data. We also experimented pre-training our model on synthetic data and fine-tuning on the real ones, although in our experience that policy led to worse results.

Visual comparison between oursreal and oursreal+synth is shown in Fig. 9. It can be seen how training only on Pascal3D+ entails artefacts for out-of-distribution viewpoints (e.g. bird’s eye view). Conversely, combining the two domains the network learns from synthetic data a prior about the overall structure and color.

We perform an analogous augmentation of the chair class. In this case we included 1858 freshly rendered synthetic images from 73 annotated models. As the rendered images are perfectly aligned with the foreground mask, this also contributes to reduce the planes’ misalignment introduced by images from Pascal3D+.

5 Conclusions

In this work we introduced a novel formulation of the problem of object novel viewpoint synthesis in a semi-parametric setting. Notably, our model is designed to be trainable on existing datasets for 3D object detection in a self-supervised manner, without the need for paired source/target viewpoint images - although it can be complemented with synthetic data. Non-parametric visual hints act as prior information to guide a deep parametric model for generating realistic images, disentangling by design appearance and shape. This enables truly continuous manipulation of the viewpoint and shape transfer to different 3D models. As completing the image is much easier than generating it from scratch, we can train our ICN on just few thousands images from the Pascal3D+ dataset and still be able to generalize to unseen viewpoints.

Although vehicles were the main focus of our work, we show that our framework is generic enough to handle rigid objects of completely different geometric structure such as chairs. Perceptual experiments results as well as image-quality metrics reward our method for its realism and the visual consistency of the synthesised object across arbitrary points of view.

Acknowledgements

This research was supported by MIUR PRIN project "PREVUE: PRediction of activities and Events by Vision in an Urban Environment", grant ID E94I19000650001.

6 Discussion on output consistency

In this work we argue that a semi-parametric method for object novel view synthesis has very appealing properties coming from the fact that the output is not generated from scratch; on the opposite, much of the information is warped from the input viewpoint in a geometrically-principled manner. We find that one of the major advantages of this semi-parametric approach is that the geometric guidance helps avoiding catastrophic failures in the generation process.

In particular, our model does not have to explicitly learn the hard concept of distance between two viewpoints in 3D space. Since the 2.5D sketch which guide the generation process are rendered from a 3D model in a purely geometric fashion, continuous change in viewpoint will lead by construction to continuous variation in the 2.5D sketch.

Conversely, in a completely learning-based pipeline the model is let to learn the concept of 3D viewpoint proximity from a massive amount of data. Still, it is a very hard concept: the model may or may not learn it properly. In Fig. 14 we make use of the VON [80] pre-trained model to showcase one situation of this kind. Even though the model is state-of-the-art and its performance is impressive, very little viewpoint variation can result in a dramatic failure in the image generation for no apparent reason.

On the opposite, the explicit texture warping and the geometric guidance given by the 2.5 sketch allows our method to produce consistent predictions from arbitrary viewpoints even tough no explicit consistency loss between different views is optimized. Visual examples from our methods are shown in Figures 15,17,16,18, where we drastically jointly change both azimuth and elevation of the target viewpoint.

Additional visual results for shape transfer on the vehicle class are depicted in Fig. 19.

7 Keypoints details

In this work semantic keypoints are leveraged as a proxy to decompose the object in the image into a small set of planar faces, with the keypoints constituting the vertices of each face.

There is no universal agreement about the number of significant objects’ landmarks, not even for most common classes such as vehicles. Indeed, different works have often used a different number of keypoints to characterize the same objects [30, 68, 67, 63, 65, 21].

Here we follow the convention of the Pascal3D+ [68] dataset, which defines 12 keypoints for vehicles (front and back wheels, upper windshield, upper rear window, front light, back trunk - left and right) and 10 for chairs (back upper, seat upper, seat lower, leg upper, leg lower - left and right).

The location of these keypoints on the Pascal3D+ models can be visually inspected in Fig. 20; some examples of annotated models are depicted in Fig. 21.

8 Planes details

Our method relies on a set of planes delimited by 2D keypoints, where each plane is defined by at least 3 keypoints. The size of this set is an hyper-parameter for our model. In fact, this value must be set accordingly to balance two factors. The first one is that, as the number of planes increases the same must hold true for the number of available keypoints. Ideally, when the number of keypoints reaches infinity, the set of planes contains the 3D surface of the object. However, the number of annotated keypoints in a dataset is seldom higher than a few dozens. On the other hand, when the number of keypoints is extremely low the set of planes hardly approximate the surface of the object. However, we found that six planes are enough for the car setting, while four suffices for the chair one.

We believe this to be due to the presence of textures and other high-frequency details on these planes. Although the side of a vehicle is not a single flat surface (thus introducing a strong approximation while relying on planar homography for the warping) the presence of those details tricks the human’s eye in believing the surface has in fact three dimensions. Then, the ICN is trained to fix inconsistencies at the borders between warped planes and to merge all the patches into a seamless figure.

9 CAD alignment details

As mentioned in experimental section of the main paper, CAD models are not perfectly aligned with images in Pascal3D+ dataset. This is due to two main reasons. Firstly, the CAD models have been placed by human workers and then refined using an automatic algorithm. Several geometric simplifications and assumptions are introduces (e.g. about camera intrinsic), and re-projection errors and human mistakes may arise. Accordingly to Pascal3D+ [68]:

The annotator (…) rotates the 3D CAD model (…). The alignment provides us with rough azimuth and elevation angles, which are used as initialization (…). We assume a simplified camera model, where the world coordinate is defined on the 3D CAD model and the camera is facing the origin of the world coordinate system (…).

Secondly, those CAD models don’t match the object portrayed, as they hail from a very reduced subset of only 10 elements. Again, from Pascal3D+ [68]:

The annotator first selects the 3D CAD model that best resembles the object instance (…).

Figure 22 shows some example of the effects of this misalignment for the chair subset, where holes and slits present an additional source of error.

10 Failure cases

We refer the reader to Fig. 23 for visual examples of failure cases for the vehicle class. In (a) the ICN is not able to recover in case of input patches which are grossly wrong, either due to a failure in keypoint estimation or to further objects occluding the car such as the people in the first image. Furthermore, we report cases in which the ICN fails to guarantee a realistic output which is consistent with the input image. This happens particularly from rarer viewpoints such as bird’s eye views (b) and from the back views (c).

Even more than for vehicles, the concave geometry of the chair object makes easier for unwanted portions of the background to leak in the generation process. In theory, our method can perfectly deal with these cases, as it masks the generated image with the silhouette of the rendered model. However, in some cases the 3D chair model rendered is so different from the one in the input image that spurious background region make their way to the final output. In Fig. 24 we provide visual examples of this case. Arguably, this problem would be much alleviated if more 3D models were used for inference (at the current state we only choose among the 10 Pascal3D+ models).

11 Visualisation Tool

We release at https://github.com/ndrplz/semiparametric the visualization tool we used to inspect the results and create most of the images in this paper. It is written in Python and depends on the Open3D [76] and OpenCV [2] libraries. The interface simulates a camera moving around an object centered in the origin. The movement are described in terms of spherical coordinates and radius, and each of the three components can be manipulated individually. The user can change:

• •

the elevation value, thus moving from lateral to bird-eye camera;

• •

the azimuth value, thus moving around the object centered in the origin describing a 360 degrees circular trajectory;

• •

the radius value, thus simulating a zoom.

The Open3D library provides a 3D environment where the CAD model is loaded, and callbacks enable rendering the scene as a 2D image. Normals visualisation is also supported. Along with the manipulation of the previous variables, the user can also independently change the CAD model and the appearance image. When started, the tool present the CAD aligned with the input image accordingly to the annotated viewpoint in the data. By changing the spherical coordinates and the radius new viewpoints of the object are synthesised, while by selecting a different CAD model a shape transfer is performed. An example of the interface is shown in 25.

12 Visual results on additional classes

To showcase the generality of our proposed approach, we report in Fig. 26 visual results on additional classes of Pascal3D+ dataset, namely Bus, Train and Sofa. The main challenge for those classes is the much lower number of images compared to the car class. Results show that our framework is able to handle a diverse set of object classes, provided that the underlying constraints (i.e. object symmetry, piece-wise planarity) hold.

13 Robustness against 2D keypoint errors

We rely on ground truth keypoints provided in the Pascal3D+ dataset for training and evaluation. Still, to assess how the model would react to keypoints predicted “in-the-wild”, we experiment with adding isotropic Gaussian noise to the 2D keypoints locations at evaluation time. Fig.27 shows how the Fréchet Inception Distance is affected by noises with increasing variance (it is worth noting that keypoints are normalized in range $[0,1]$). Although the results obviously degrade with respect to the annotated keypoints locations, we notice that overall performance is still competitive against most competitors even when adding aggressive noises.

14 Additional ablation: only appearance prior

Figure 28 depicts visual results for a network trained using the appearance prior only. While the general color from the central crop is applied to the destination image, almost all other details are lacking. This emphasises the importance of the image planes as the main source of texture in the final image. The Fréchet Inception Distance score for this ablated version (averaged across all 12 angles) is 154.2.

15 A/B Perceptual Experiments

We developed a tool to perform A/B test preferences based on Django web server. The system is centralised and can be accessed from multiple terminals simultaneously. We wrote asking for volunteers on the University of Modena and Reggio Emilia mailing list, and the tests were performed from the volunteer’s computer. We didn’t interact directly with participants to avoid introducing any bias inadvertently. To ensure the correctness of the data we discarded the first results from each session, which were considered as a briefing. For all experiments, images were shown at the same resolution of 128x128 pixels. As all methods produce a white background, ground truth aligned 3D CAD is used to mask Pascal3D+ real images. Both sampling order and left-right order of A and B were randomized. Raw results have been stored into a MySQL DB to ensure persistence, while aggregated statistics were presented on an administrator interface. Fig. 16 depicts screenshots from the two perceptual experiments described in Sec.4 of the main paper.