Anytime 3D Object Reconstruction using Multi-modal Variational Autoencoder

TL;DR

This paper introduces a multi-modal variational autoencoder that enables real-time, anytime 3D object reconstruction from incomplete data, improving robustness and efficiency in remote human-robot collaboration.

Contribution

It proposes a category-specific multi-modal prior in the latent space to improve imputation and reconstruction from partial data, surpassing traditional autoencoders.

Findings

Outperforms standard autoencoder and VAE with up to 70% data loss.

Demonstrates effective 3D reconstruction on ModelNet and Pascal3D datasets.

Enables data over-compression and partial data imputation.

Abstract

For effective human-robot teaming, it is important for the robots to be able to share their visual perception with the human operators. In a harsh remote collaboration setting, data compression techniques such as autoencoder can be utilized to obtain and transmit the data in terms of latent variables in a compact form. In addition, to ensure real-time runtime performance even under unstable environments, an anytime estimation approach is desired that can reconstruct the full contents from incomplete information. In this context, we propose a method for imputation of latent variables whose elements are partially lost. To achieve the anytime property with only a few dimensions of variables, exploiting prior information of the category-level is essential. A prior distribution used in variational autoencoders is simply assumed to be isotropic Gaussian regardless of the labels of each training datapoint. This type of flattened prior makes it difficult to perform imputation from the category-level distributions. We overcome this limitation by exploiting a category-specific multi-modal prior distribution in the latent space. The missing elements of the partially transferred data can be sampled, by finding a specific modal according to the remaining elements. Since the method is designed to use partial elements for anytime estimation, it can also be applied for data over-compression. Based on the experiments on the ModelNet and Pascal3D datasets, the proposed approach shows consistently superior performance over autoencoder and variational autoencoder up to 70% data loss.