SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data

TL;DR

This paper introduces SAIL-VOS 3D, a synthetic video dataset with mesh annotations and baseline models for 3D object reconstruction from video, demonstrating that temporal data enhances reconstruction accuracy.

Contribution

The paper presents the first synthetic video dataset with mesh annotations for 3D reconstruction and develops baseline temporal models to improve mesh reconstruction from videos.

Findings

Temporal models outperform single-image methods.

Using video data improves 3D mesh reconstruction quality.

SAIL-VOS 3D dataset enables studying temporal effects in 3D reconstruction.

Abstract

Extracting detailed 3D information of objects from video data is an important goal for holistic scene understanding. While recent methods have shown impressive results when reconstructing meshes of objects from a single image, results often remain ambiguous as part of the object is unobserved. Moreover, existing image-based datasets for mesh reconstruction don't permit to study models which integrate temporal information. To alleviate both concerns we present SAIL-VOS 3D: a synthetic video dataset with frame-by-frame mesh annotations which extends SAIL-VOS. We also develop first baselines for reconstruction of 3D meshes from video data via temporal models. We demonstrate efficacy of the proposed baseline on SAIL-VOS 3D and Pix3D, showing that temporal information improves reconstruction quality. Resources and additional information are available at http://sailvos.web.illinois.edu.