Learning a Predictable and Generative Vector Representation for Objects

TL;DR

This paper introduces a novel vector embedding for objects that is both generative in 3D and predictable from 2D images, enabling various 3D understanding tasks.

Contribution

The paper proposes the TL-embedding network, combining autoencoding and convolutional prediction to create a versatile 3D object representation.

Findings

Effective voxel prediction from 2D images

Improved 3D model retrieval performance

Versatile application across 3D understanding tasks

Abstract

What is a good vector representation of an object? We believe that it should be generative in 3D, in the sense that it can produce new 3D objects; as well as be predictable from 2D, in the sense that it can be perceived from 2D images. We propose a novel architecture, called the TL-embedding network, to learn an embedding space with these properties. The network consists of two components: (a) an autoencoder that ensures the representation is generative; and (b) a convolutional network that ensures the representation is predictable. This enables tackling a number of tasks including voxel prediction from 2D images and 3D model retrieval. Extensive experimental analysis demonstrates the usefulness and versatility of this embedding.