Semantic Aware Feature Extraction for Enhanced 3D Reconstruction

TL;DR

This paper introduces a semantic-aware feature extraction framework that combines multi-task learning and deep matching to improve 3D reconstruction quality by incorporating semantic and elevation information from monocular fisheye camera data.

Contribution

It presents a novel joint training approach for keypoint detection, description, and semantic segmentation to enhance feature matching and 3D reconstruction accuracy.

Findings

Improved structural detail in 3D point clouds.

Enhanced elevation estimation and multi-level mapping.

Better feature correspondence through semantic cues.

Abstract

Feature matching is a fundamental problem in computer vision with wide-ranging applications, including simultaneous localization and mapping (SLAM), image stitching, and 3D reconstruction. While recent advances in deep learning have improved keypoint detection and description, most approaches focus primarily on geometric attributes and often neglect higher-level semantic information. This work proposes a semantic-aware feature extraction framework that employs multi-task learning to jointly train keypoint detection, keypoint description, and semantic segmentation. The method is benchmarked against standard feature matching techniques and evaluated in the context of 3D reconstruction. To enhance feature correspondence, a deep matching module is integrated. The system is tested using input from a single monocular fisheye camera mounted on a vehicle and evaluated within a multi-floor parking structure. The proposed approach supports semantic 3D reconstruction with altitude estimation, capturing elevation changes and enabling multi-level mapping. Experimental results demonstrate that the method produces semantically annotated 3D point clouds with improved structural detail and elevation information, underscoring the effectiveness of joint training with semantic cues for more consistent feature matching and enhanced 3D reconstruction.