SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains

TL;DR

SAGE is a style-adaptive framework that enhances domain generalization for semantic segmentation by synthesizing visual prompts, without modifying model weights, to better handle unseen styles under privacy constraints.

Contribution

SAGE introduces a novel input-level style synthesis approach that improves generalization of frozen models across domains without requiring access to model parameters.

Findings

SAGE outperforms state-of-the-art methods on five benchmark datasets.

SAGE surpasses full fine-tuning baselines in all tested scenarios.

SAGE effectively bridges the gap between model invariance and domain diversity.

Abstract

Domain generalization for semantic segmentation aims to mitigate the degradation in model performance caused by domain shifts. However, in many real-world scenarios, we are unable to access the model parameters and architectural details due to privacy concerns and security constraints. Traditional fine-tuning or adaptation is hindered, leading to the demand for input-level strategies that can enhance generalization without modifying model weights. To this end, we propose a \textbf{S}tyle-\textbf{A}daptive \textbf{GE}neralization framework (\textbf{SAGE}), which improves the generalization of frozen models under privacy constraints. SAGE learns to synthesize visual prompts that implicitly align feature distributions across styles instead of directly fine-tuning the backbone. Specifically, we first utilize style transfer to construct a diverse style representation of the source domain, thereby learning a set of style characteristics that can cover a wide range of visual features. Then, the model adaptively fuses these style cues according to the visual context of each input, forming a dynamic prompt that harmonizes the image appearance without touching the interior of the model. Through this closed-loop design, SAGE effectively bridges the gap between frozen model invariance and the diversity of unseen domains. Extensive experiments on five benchmark datasets demonstrate that SAGE achieves competitive or superior performance compared to state-of-the-art methods under privacy constraints and outperforms full fine-tuning baselines in all settings.