# Semantically Informed Multiview Surface Refinement

## TL;DR

This paper introduces a joint method for refining 3D surface geometry and semantic segmentation by alternating shape updates with semantic label adjustments, leveraging priors and multi-view consistency.

## Contribution

It presents a novel mesh-based approach that tightly couples geometry and semantics through priors and iterative refinement, outperforming voxel-based and purely geometric methods.

## Key findings

- Improved 3D geometry accuracy over state-of-the-art methods.
- Enhanced semantic segmentation quality on 3D meshes.
- Effective integration of semantic priors into surface refinement.

## Abstract

We present a method to jointly refine the geometry and semantic segmentation of 3D surface meshes. Our method alternates between updating the shape and the semantic labels. In the geometry refinement step, the mesh is deformed with variational energy minimization, such that it simultaneously maximizes photo-consistency and the compatibility of the semantic segmentations across a set of calibrated images. Label-specific shape priors account for interactions between the geometry and the semantic labels in 3D. In the semantic segmentation step, the labels on the mesh are updated with MRF inference, such that they are compatible with the semantic segmentations in the input images. Also, this step includes prior assumptions about the surface shape of different semantic classes. The priors induce a tight coupling, where semantic information influences the shape update and vice versa. Specifically, we introduce priors that favor (i) adaptive smoothing, depending on the class label; (ii) straightness of class boundaries; and (iii) semantic labels that are consistent with the surface orientation. The novel mesh-based reconstruction is evaluated in a series of experiments with real and synthetic data. We compare both to state-of-the-art, voxel-based semantic 3D reconstruction, and to purely geometric mesh refinement, and demonstrate that the proposed scheme yields improved 3D geometry as well as an improved semantic segmentation.