Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification

TL;DR

This paper introduces a novel multi-modal skin cancer classification approach combining dermatological images and patient metadata using a joint-individual fusion structure and a fusion attention module, leading to improved accuracy.

Contribution

The paper proposes a new fusion method with a joint-individual structure and attention mechanism, enhancing multi-modal skin cancer classification accuracy over existing methods.

Findings

Outperforms state-of-the-art fusion methods on three datasets

Improves classification accuracy across multiple CNN backbones

Demonstrates robustness and effectiveness of the proposed approach

Abstract

Most convolutional neural network (CNN) based methods for skin cancer classification obtain their results using only dermatological images. Although good classification results have been shown, more accurate results can be achieved by considering the patient's metadata, which is valuable clinical information for dermatologists. Current methods only use the simple joint fusion structure (FS) and fusion modules (FMs) for the multi-modal classification methods, there still is room to increase the accuracy by exploring more advanced FS and FM. Therefore, in this paper, we design a new fusion method that combines dermatological images (dermoscopy images or clinical images) and patient metadata for skin cancer classification from the perspectives of FS and FM. First, we propose a joint-individual fusion (JIF) structure that learns the shared features of multi-modality data and preserves specific features simultaneously. Second, we introduce a fusion attention (FA) module that enhances the most relevant image and metadata features based on both the self and mutual attention mechanism to support the decision-making pipeline. We compare the proposed JIF-MMFA method with other state-of-the-art fusion methods on three different public datasets. The results show that our JIF-MMFA method improves the classification results for all tested CNN backbones and performs better than the other fusion methods on the three public datasets, demonstrating our method's effectiveness and robustness