Language-Driven Dual Style Mixing for Single-Domain Generalized Object Detection

TL;DR

This paper introduces a novel language-driven dual style mixing approach for single-domain object detection that enhances generalization to unseen domains by leveraging semantic information from vision-language models for both image and feature augmentation.

Contribution

The proposed LDDS method utilizes semantic prompts from VLMs to generate style-diversified images and feature augmentations, enabling model-agnostic domain generalization without specific augmentation choices.

Findings

Improves detection performance across various unseen domains.

Effective in tasks like real to cartoon and normal to adverse weather.

Compatible with multiple mainstream detector frameworks.

Abstract

Generalizing an object detector trained on a single domain to multiple unseen domains is a challenging task. Existing methods typically introduce image or feature augmentation to diversify the source domain to raise the robustness of the detector. Vision-Language Model (VLM)-based augmentation techniques have been proven to be effective, but they require that the detector's backbone has the same structure as the image encoder of VLM, limiting the detector framework selection. To address this problem, we propose Language-Driven Dual Style Mixing (LDDS) for single-domain generalization, which diversifies the source domain by fully utilizing the semantic information of the VLM. Specifically, we first construct prompts to transfer style semantics embedded in the VLM to an image translation network. This facilitates the generation of style diversified images with explicit semantic information. Then, we propose image-level style mixing between the diversified images and source domain images. This effectively mines the semantic information for image augmentation without relying on specific augmentation selections. Finally, we propose feature-level style mixing in a double-pipeline manner, allowing feature augmentation to be model-agnostic and can work seamlessly with the mainstream detector frameworks, including the one-stage, two-stage, and transformer-based detectors. Extensive experiments demonstrate the effectiveness of our approach across various benchmark datasets, including real to cartoon and normal to adverse weather tasks. The source code and pre-trained models will be publicly available at https://github.com/qinhongda8/LDDS.