Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance

TL;DR

ST-STORM is a hybrid self-supervised learning framework that disentangles appearance and content, improving semantic robustness and appearance understanding in tasks like weather analysis and medical imaging.

Contribution

It introduces a dual-stream architecture with gating mechanisms to separately learn appearance signatures and invariant semantic representations.

Findings

Style branch captures complex appearance phenomena with high F1 scores (97% on Multi-Weather, 94% on ISIC 2024).

Content branch maintains semantic performance with 80% F1 on ImageNet-1K.

Disentangling appearance improves critical task performance without degrading semantic accuracy.

Abstract

One of the dominant paradigms in self-supervised learning (SSL), illustrated by MoCo or DINO, aims to produce robust representations by capturing features that are insensitive to certain image transformations such as illumination, or geometric changes. This strategy is appropriate when the objective is to recognize objects independently of their appearance. However, it becomes counterproductive as soon as appearance itself constitutes the discriminative signal. In weather analysis, for example, rain streaks, snow granularity, atmospheric scattering, as well as reflections and halos, are not noise: they carry the essential information. In critical applications such as autonomous driving, ignoring these cues is risky, since grip and visibility depend directly on ground conditions and atmospheric conditions. We introduce ST-STORM, a hybrid SSL framework that treats appearance (style) as a semantic modality to be disentangled from content. Our architecture explicitly separates two latent streams, regulated by gating mechanisms. The Content branch aims at a stable semantic representation through a JEPA scheme coupled with a contrastive objective, promoting invariance to appearance variations. In parallel, the Style branch is constrained to capture appearance signatures (textures, contrasts, scattering) through feature prediction and reconstruction under an adversarial constraint. We evaluate ST-STORM on several tasks, including object classification (ImageNet-1K), fine-grained weather characterization, and melanoma detection (ISIC 2024 Challenge). The results show that the Style branch effectively isolates complex appearance phenomena (F1=97% on Multi-Weather and F1=94% on ISIC 2024 with 10% labeled data), without degrading the semantic performance (F1=80% on ImageNet-1K) of the Content branch, and improves the preservation of critical appearance