# Multimodal Machine Learning Approach for Diagnosing Atopic Dermatitis

## TL;DR

A new AI model combines skin images and patient history to diagnose atopic dermatitis with high accuracy, potentially improving consistency and reducing subjectivity in diagnosis.

## Contribution

A novel multimodal AI model integrating lesion images and anamnesis data achieves high diagnostic accuracy for atopic dermatitis.

## Key findings

- The multimodal AI model achieved 98.28% accuracy in classifying AD vs non-AD.
- The model outperformed image-only or text-only models in diagnostic performance.
- The model mimics clinical reasoning by combining visual and contextual information.

## Abstract

Atopic dermatitis (AD) is a prevalent, chronic inflammatory skin disease with diverse clinical presentations, often overlapping with other dermatoses. Its diagnosis remains largely dependent on clinical expertise, leading to variability and limited diagnostic accuracy, particularly among general practitioners. This study aimed to develop and evaluate a multimodal artificial intelligence (AI) model that integrates lesion image analysis and structured anamnesis to improve AD diagnosis.

This diagnostic study was conducted in two phases: Phase 1 used retrospective data from 2021–2024, and Phase 2 involved prospective external validation from multiple hospitals in 2025. Patients with AD or related skin conditions were included, with diagnoses based on AAD 2014 criteria. Multimodal fusion combined ResNet50-extracted image features and MPNet-based anamnesis text features using a late fusion model. This approach mimics clinical reasoning by integrating visual and contextual clinical information to classify cases as AD or non-AD.

The multimodal AI model integrating ResNet50 (image) and MPNet (anamnesis) achieved 98.28% accuracy in classifying AD vs non-AD, outperforming image-or text-only models. It offers clinical advantages by mimicking physician reasoning, improving diagnostic consistency, reducing subjectivity, and enabling mass triage. However, real-world generalizability remains a challenge due to limited training diversity, potential language constraints (Bahasa Indonesia), and narrow differential diagnoses. External validation and explainable AI (XAI) are critical for broader application. Despite limitations, the model aligns with emerging literature, showing multimodal AI can approach or surpass expert-level performance in dermatological diagnosis when rigorously validated.

The multimodal ResNet50-MPNet model shows near-perfect accuracy in diagnosing AD by mimicking clinician reasoning. It offers consistent, holistic assessment but requires external validation and improved interpretability for clinical adoption. Continued AI-clinician collaboration is vital to translating this promising technology into real-world dermatological care.