Pay Less On Clinical Images: Asymmetric Multi-Modal Fusion Method For Efficient Multi-Label Skin Lesion Classification

TL;DR

This paper introduces an efficient asymmetric multi-modal fusion method for multi-label skin lesion classification, utilizing a lightweight clinical image network and a complex dermoscopy network with an asymmetric attention module, reducing parameters while improving performance.

Contribution

The paper proposes a novel asymmetric fusion structure and attention mechanism that leverage clinical images as supplementary information, significantly reducing parameters compared to symmetric methods.

Findings

Parameter savings over symmetric fusion structures

Effective enhancement of dermoscopy features using clinical images

Superior performance on the seven-point checklist dataset

Abstract

Existing multi-modal approaches primarily focus on enhancing multi-label skin lesion classification performance through advanced fusion modules, often neglecting the associated rise in parameters. In clinical settings, both clinical and dermoscopy images are captured for diagnosis; however, dermoscopy images exhibit more crucial visual features for multi-label skin lesion classification. Motivated by this observation, we introduce a novel asymmetric multi-modal fusion method in this paper for efficient multi-label skin lesion classification. Our fusion method incorporates two innovative schemes. Firstly, we validate the effectiveness of our asymmetric fusion structure. It employs a light and simple network for clinical images and a heavier, more complex one for dermoscopy images, resulting in significant parameter savings compared to the symmetric fusion structure using two identical networks for both modalities. Secondly, in contrast to previous approaches using mutual attention modules for interaction between image modalities, we propose an asymmetric attention module. This module solely leverages clinical image information to enhance dermoscopy image features, considering clinical images as supplementary information in our pipeline. We conduct the extensive experiments on the seven-point checklist dataset. Results demonstrate the generality of our proposed method for both networks and Transformer structures, showcasing its superiority over existing methods We will make our code publicly available.