Learning to Reconstruct and Segment 3D Objects

TL;DR

This paper explores deep learning methods for reconstructing and segmenting 3D objects from various visual inputs, aiming to improve understanding of 3D environments beyond traditional hand-crafted approaches.

Contribution

It introduces novel deep neural network techniques for 3D shape estimation and semantic segmentation at object and scene levels, trained on large-scale real-world data.

Findings

Deep neural networks outperform traditional methods in 3D shape estimation.

Robust segmentation achieved even with occlusions and limited views.

Scene understanding improved through integrated object-level analysis.

Abstract

To endow machines with the ability to perceive the real-world in a three dimensional representation as we do as humans is a fundamental and long-standing topic in Artificial Intelligence. Given different types of visual inputs such as images or point clouds acquired by 2D/3D sensors, one important goal is to understand the geometric structure and semantics of the 3D environment. Traditional approaches usually leverage hand-crafted features to estimate the shape and semantics of objects or scenes. However, they are difficult to generalize to novel objects and scenarios, and struggle to overcome critical issues caused by visual occlusions. By contrast, we aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks, trained on large-scale real-world 3D data. To achieve these aims, this thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding.