Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation

TL;DR

This paper introduces MADM, a novel approach using text-to-image diffusion models for unsupervised multimodal domain adaptation in semantic segmentation, improving label accuracy and feature resolution across various modalities.

Contribution

MADM leverages diffusion models for pseudo-label generation and introduces label palette and latent regression techniques to enhance cross-modality semantic segmentation.

Findings

Achieves state-of-the-art results on multiple modality adaptation tasks

Improves pseudo-label accuracy with diffusion-based noise stabilization

Enhances feature resolution through label palette and latent regression

Abstract

Despite their success, unsupervised domain adaptation methods for semantic segmentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components to tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modality suffers from a significant drop in accuracy. To address this, MADM designs diffusion-based pseudo-label generation which adds latent noise to stabilize pseudo-labels and enhance label accuracy. Second, to overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling to obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at https://github.com/XiaRho/MADM.