Semantic-Space-Intervened Diffusive Alignment for Visual Classification

TL;DR

This paper introduces SeDA, a novel diffusion-based method that progressively aligns visual and textual features via a shared semantic space, significantly improving cross-modal visual classification accuracy.

Contribution

SeDA employs a bi-stage diffusion framework with a semantic space bridge and stepwise feature interaction, advancing cross-modal alignment techniques.

Findings

SeDA outperforms existing methods in multiple visual classification scenarios.

The diffusion-controlled semantic models improve cross-modal feature consistency.

Progressive alignment enhances the integration of textual information into visual features.

Abstract

Cross-modal alignment is an effective approach to improving visual classification. Existing studies typically enforce a one-step mapping that uses deep neural networks to project the visual features to mimic the distribution of textual features. However, they typically face difficulties in finding such a projection due to the two modalities in both the distribution of class-wise samples and the range of their feature values. To address this issue, this paper proposes a novel Semantic-Space-Intervened Diffusive Alignment method, termed SeDA, models a semantic space as a bridge in the visual-to-textual projection, considering both types of features share the same class-level information in classification. More importantly, a bi-stage diffusion framework is developed to enable the progressive alignment between the two modalities. Specifically, SeDA first employs a Diffusion-Controlled Semantic Learner to model the semantic features space of visual features by constraining the interactive features of the diffusion model and the category centers of visual features. In the later stage of SeDA, the Diffusion-Controlled Semantic Translator focuses on learning the distribution of textual features from the semantic space. Meanwhile, the Progressive Feature Interaction Network introduces stepwise feature interactions at each alignment step, progressively integrating textual information into mapped features. Experimental results show that SeDA achieves stronger cross-modal feature alignment, leading to superior performance over existing methods across multiple scenarios.