Semantic Embedded Deep Neural Network: A Generic Approach to Boost Multi-Label Image Classification Performance

TL;DR

This paper proposes a semantic-embedding deep neural network with spatial awareness and channel-wise attention to improve multi-label image classification, especially in cluttered backgrounds, demonstrating significant performance gains.

Contribution

Introduces a generic semantic-embedding deep neural network with spatial and attention mechanisms to enhance multi-label classification accuracy in complex visual environments.

Findings

15.27% average relative improvement in AUC score

Effective localization guidance improves multi-label prediction

Outperforms baseline and alternative semantic feature methods

Abstract

Fine-grained multi-label classification models have broad applications in e-commerce, such as visual based label predictions ranging from fashion attribute detection to brand recognition. One challenge to achieve satisfactory performance for those classification tasks in real world is the wild visual background signal that contains irrelevant pixels which confuses model to focus onto the region of interest and make prediction upon the specific region. In this paper, we introduce a generic semantic-embedding deep neural network to apply the spatial awareness semantic feature incorporating a channel-wise attention based model to leverage the localization guidance to boost model performance for multi-label prediction. We observed an Avg.relative improvement of 15.27% in terms of AUC score across all labels compared to the baseline approach. Core experiment and ablation studies involve multi-label fashion attribute classification performed on Instagram fashion apparels' image. We compared the model performances among our approach, baseline approach, and 3 alternative approaches to leverage semantic features. Results show favorable performance for our approach.