Object Detection in the Context of Mobile Augmented Reality

TL;DR

This paper introduces a novel method that combines Visual-Inertial Odometry with semantic object detection to enhance real-time object detection accuracy in mobile augmented reality applications.

Contribution

It presents a new approach integrating VIO with object detectors, including orientation correction, scale filtering, and semantic mapping, tailored for mobile AR devices.

Findings

Improved object detection accuracy by 12% on AR datasets.

Effective integration of VIO and semantic information for AR.

Validated on Android with SSD MobileNet and ARCore.

Abstract

In the past few years, numerous Deep Neural Network (DNN) models and frameworks have been developed to tackle the problem of real-time object detection from RGB images. Ordinary object detection approaches process information from the images only, and they are oblivious to the camera pose with regard to the environment and the scale of the environment. On the other hand, mobile Augmented Reality (AR) frameworks can continuously track a camera's pose within the scene and can estimate the correct scale of the environment by using Visual-Inertial Odometry (VIO). In this paper, we propose a novel approach that combines the geometric information from VIO with semantic information from object detectors to improve the performance of object detection on mobile devices. Our approach includes three components: (1) an image orientation correction method, (2) a scale-based filtering approach, and (3) an online semantic map. Each component takes advantage of the different characteristics of the VIO-based AR framework. We implemented the AR-enhanced features using ARCore and the SSD Mobilenet model on Android phones. To validate our approach, we manually labeled objects in image sequences taken from 12 room-scale AR sessions. The results show that our approach can improve on the accuracy of generic object detectors by 12% on our dataset.