Contextually Guided Semantic Labeling and Search for 3D Point Clouds

TL;DR

This paper introduces a graphical model-based method for semantic labeling of 3D point clouds from RGB-D data, leveraging contextual relations to improve object detection and enabling robots to search for objects effectively.

Contribution

It presents a novel graphical model incorporating multiple contextual cues and a maximum-margin training approach for improved semantic labeling and object search in 3D scenes.

Findings

Achieved 84.06% accuracy in office scenes and 73.38% in home scenes for 17 object classes.

Successfully applied the object search method on a robot with 97.56% precision.

Demonstrated effective scene labeling and object search in real-world indoor environments.

Abstract

RGB-D cameras, which give an RGB image to- gether with depths, are becoming increasingly popular for robotic perception. In this paper, we address the task of detecting commonly found objects in the 3D point cloud of indoor scenes obtained from such cameras. Our method uses a graphical model that captures various features and contextual relations, including the local visual appearance and shape cues, object co-occurence relationships and geometric relationships. With a large number of object classes and relations, the model's parsimony becomes important and we address that by using multiple types of edge potentials. We train the model using a maximum-margin learning approach. In our experiments over a total of 52 3D scenes of homes and offices (composed from about 550 views), we get a performance of 84.06% and 73.38% in labeling office and home scenes respectively for 17 object classes each. We also present a method for a robot to search for an object using the learned model and the contextual information available from the current labelings of the scene. We applied this algorithm successfully on a mobile robot for the task of finding 12 object classes in 10 different offices and achieved a precision of 97.56% with 78.43% recall.