View Generalization for Single Image Textured 3D Models

TL;DR

This paper introduces a new model class with controllable geometric rigidity and cycle consistency losses to improve view generalization in single-image textured 3D model inference, outperforming existing methods.

Contribution

It proposes a novel approach combining geometric rigidity control and cycle consistency losses to enhance view generalization and texture sharing in 3D model inference from a single image.

Findings

Significant improvement in view generalization over state-of-the-art methods.

Enhanced texture sharing across different views.

Quantitative and qualitative validation of the proposed approach.

Abstract

Humans can easily infer the underlying 3D geometry and texture of an object only from a single 2D image. Current computer vision methods can do this, too, but suffer from view generalization problems - the models inferred tend to make poor predictions of appearance in novel views. As for generalization problems in machine learning, the difficulty is balancing single-view accuracy (cf. training error; bias) with novel view accuracy (cf. test error; variance). We describe a class of models whose geometric rigidity is easily controlled to manage this tradeoff. We describe a cycle consistency loss that improves view generalization (roughly, a model from a generated view should predict the original view well). View generalization of textures requires that models share texture information, so a car seen from the back still has headlights because other cars have headlights. We describe a cycle consistency loss that encourages model textures to be aligned, so as to encourage sharing. We compare our method against the state-of-the-art method and show both qualitative and quantitative improvements.