3D Annotation Of Arbitrary Objects In The Wild

TL;DR

This paper introduces a novel annotation pipeline that leverages SLAM, 3D reconstruction, and geometry to efficiently annotate arbitrary objects in real environments, significantly reducing manual effort and enabling better domain-specific data collection for deep learning.

Contribution

The proposed pipeline allows for fast, accurate 3D and 2D annotations of objects without requiring pre-existing 3D models, facilitating domain-specific data collection for robotic applications.

Findings

Achieved nearly 90% IoU in segmentation and detection tasks.

Significantly faster annotation process compared to manual methods.

Applicable to diverse objects and scenes.

Abstract

Recent years have produced a variety of learning based methods in the context of computer vision and robotics. Most of the recently proposed methods are based on deep learning, which require very large amounts of data compared to traditional methods. The performance of the deep learning methods are largely dependent on the data distribution they were trained on, and it is important to use data from the robot's actual operating domain during training. Therefore, it is not possible to rely on pre-built, generic datasets when deploying robots in real environments, creating a need for efficient data collection and annotation in the specific operating conditions the robots will operate in. The challenge is then: how do we reduce the cost of obtaining such datasets to a point where we can easily deploy our robots in new conditions, environments and to support new sensors? As an answer to this question, we propose a data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry. The pipeline allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects without needing accurate 3D models of the objects prior to data collection and annotation. Our results showcase almost 90% Intersection-over-Union (IoU) agreement on both semantic segmentation and 2D bounding box detection across a variety of objects and scenes, while speeding up the annotation process by several orders of magnitude compared to traditional manual annotation.