Shooting Labels: 3D Semantic Labeling by Virtual Reality

TL;DR

This paper introduces Shooting Labels, a virtual reality-based tool for rapid, accurate 3D semantic labeling that leverages multiuser input and can project annotations into 2D images, streamlining dataset creation.

Contribution

It presents the first VR-based 3D labeling tool that simplifies dense semantic segmentation and integrates multiuser annotations to enhance accuracy and efficiency.

Findings

Effective labeling of large-scale environments demonstrated

Multiuser annotations improve accuracy automatically

Annotations can be projected into 2D images for faster labeling

Abstract

Availability of a few, large-size, annotated datasets, like ImageNet, Pascal VOC and COCO, has lead deep learning to revolutionize computer vision research by achieving astonishing results in several vision tasks.We argue that new tools to facilitate generation of annotated datasets may help spreading data-driven AI throughout applications and domains. In this work we propose Shooting Labels, the first 3D labeling tool for dense 3D semantic segmentation which exploits Virtual Reality to render the labeling task as easy and fun as playing a video-game. Our tool allows for semantically labeling large scale environments very expeditiously, whatever the nature of the 3D data at hand (e.g. point clouds, mesh). Furthermore, Shooting Labels efficiently integrates multiusers annotations to improve the labeling accuracy automatically and compute a label uncertainty map. Besides, within our framework the 3D annotations can be projected into 2D images, thereby speeding up also a notoriously slow and expensive task such as pixel-wise semantic labeling. We demonstrate the accuracy and efficiency of our tool in two different scenarios: an indoor workspace provided by Matterport3D and a large-scale outdoor environment reconstructed from 1000+ KITTI images.