Depth-guided Texture Diffusion for Image Semantic Segmentation

TL;DR

This paper introduces a Depth-guided Texture Diffusion method that enhances depth maps with texture information to improve semantic segmentation accuracy across various datasets, effectively bridging the modality gap.

Contribution

The proposed approach uniquely diffuses texture information into depth maps to better integrate depth and RGB features for semantic segmentation.

Findings

Outperforms existing baselines on multiple datasets

Achieves state-of-the-art results with estimated and captured depth

Effectively enhances structural details for segmentation

Abstract

Depth information provides valuable insights into the 3D structure especially the outline of objects, which can be utilized to improve the semantic segmentation tasks. However, a naive fusion of depth information can disrupt feature and compromise accuracy due to the modality gap between the depth and the vision. In this work, we introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. This image is then selectively diffused across the depth map, enhancing structural information vital for precisely extracting object outlines. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image, enabling more accurate semantic segmentation. We conduct comprehensive experiments across diverse, commonly-used datasets spanning a wide range of semantic segmentation tasks, including Camouflaged Object Detection (COD), Salient Object Detection (SOD), and indoor semantic segmentation. With source-free estimated depth or depth captured by depth cameras, our method consistently outperforms existing baselines and achieves new state-of-theart results, demonstrating the effectiveness of our Depth-guided Texture Diffusion for image semantic segmentation.