Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans

TL;DR

This paper presents Omnidata, a scalable pipeline that generates multi-task vision datasets from 3D scans, enabling training of robust models that achieve state-of-the-art results on various benchmarks.

Contribution

The authors introduce a parametrically controllable pipeline for creating multi-task datasets from 3D scans, facilitating research and improving model performance.

Findings

Models trained on generated data achieved state-of-the-art results.

Depth estimation network outperforms MiDaS.

Surface normal estimation reaches human-level performance.

Abstract

This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. In addition to enabling interesting lines of research, we show the tooling and generated data suffice to train robust vision models. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks, despite having seen no benchmark or non-pipeline data. The depth estimation network outperforms MiDaS and the surface normal estimation network is the first to achieve human-level performance for in-the-wild surface normal estimation -- at least according to one metric on the OASIS benchmark. The Dockerized pipeline with CLI, the (mostly python) code, PyTorch dataloaders for the generated data, the generated starter dataset, download scripts and other utilities are available through our project website, https://omnidata.vision.