A 3D-Deep-Learning-based Augmented Reality Calibration Method for Robotic Environments using Depth Sensor Data

TL;DR

This paper presents a novel 3D deep learning method for calibrating augmented reality devices with robotic environments using depth sensor data, eliminating the need for external markers and enhancing flexibility.

Contribution

It introduces a deep learning calibration approach based on VoteNet architecture for AR devices using depth sensors, and provides an open source point cloud labeling tool.

Findings

Achieved effective calibration without external markers

Method adaptable to various depth cameras

Open source tool for point cloud labeling

Abstract

Augmented Reality and mobile robots are gaining much attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this account, we propose a novel approach to calibrate the Augmented Reality device using 3D depth sensor data. We use the depth camera of a cutting edge Augmented Reality Device - the Microsoft Hololens for deep learning based calibration. Therefore, we modified a neural network based on the recently published VoteNet architecture which works directly on the point cloud input observed by the Hololens. We achieve satisfying results and eliminate external tools like markers, thus enabling a more intuitive and flexible work flow for Augmented Reality integration. The results are adaptable to work with all depth cameras and are promising for further research. Furthermore, we introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.