A Simple Recipe for Language-guided Domain Generalized Segmentation

TL;DR

This paper proposes a straightforward method for improving semantic segmentation generalization to new domains by leveraging language-guided style augmentation and minimal fine-tuning of CLIP, achieving state-of-the-art results.

Contribution

It introduces a simple, effective framework that uses language as a source of randomization for domain generalization in segmentation tasks, with minimal fine-tuning of CLIP.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Effective use of language-guided style augmentation.

Minimal fine-tuning preserves CLIP robustness.

Abstract

Generalization to new domains not seen during training is one of the long-standing challenges in deploying neural networks in real-world applications. Existing generalization techniques either necessitate external images for augmentation, and/or aim at learning invariant representations by imposing various alignment constraints. Large-scale pretraining has recently shown promising generalization capabilities, along with the potential of binding different modalities. For instance, the advent of vision-language models like CLIP has opened the doorway for vision models to exploit the textual modality. In this paper, we introduce a simple framework for generalizing semantic segmentation networks by employing language as the source of randomization. Our recipe comprises three key ingredients: (i) the preservation of the intrinsic CLIP robustness through minimal fine-tuning, (ii) language-driven local style augmentation, and (iii) randomization by locally mixing the source and augmented styles during training. Extensive experiments report state-of-the-art results on various generalization benchmarks. Code is accessible at https://github.com/astra-vision/FAMix .