Object Localization and Size Estimation from RGB-D Images

TL;DR

This paper presents a method for localizing objects and estimating their real-world size using RGB-D images from a Tango phone, combining color and depth data for improved segmentation and measurement.

Contribution

It introduces a process for collecting, aligning, and analyzing RGB-D data from a mobile device to enhance object localization and size estimation.

Findings

Depth data improves object segmentation accuracy.

Real-world size estimation is feasible with aligned RGB-D data.

The approach works under various environmental conditions.

Abstract

Depth sensing cameras (e.g., Kinect sensor, Tango phone) can acquire color and depth images that are registered to a common viewpoint. This opens the possibility of developing algorithms that exploit the advantages of both sensing modalities. Traditionally, cues from color images have been used for object localization (e.g., YOLO). However, the addition of a depth image can be further used to segment images that might otherwise have identical color information. Further, the depth image can be used for object size (height/width) estimation (in real-world measurements units, such as meters) as opposed to image based segmentation that would only support drawing bounding boxes around objects of interest. In this paper, we first collect color camera information along with depth information using a custom Android application on Tango Phab2 phone. Second, we perform timing and spatial alignment between the two data sources. Finally, we evaluate several ways of measuring the height of the object of interest within the captured images under a variety of settings.