Sumotosima: A Framework and Dataset for Classifying and Summarizing Otoscopic Images

TL;DR

Sumotosima is a novel deep learning framework that classifies otoscopic images and generates patient-friendly summaries, utilizing a new dataset and outperforming existing models in accuracy and summarization quality.

Contribution

The paper introduces Sumotosima, a resource-efficient transformer-based framework with a new dataset for classifying and summarizing otoscopic images, advancing diagnostic support tools.

Findings

Achieved 98.03% classification accuracy, surpassing traditional models.

Outperformed GPT-4o and LLaVA in ROUGE scores for summarization.

Curated the OCASD dataset with 500 expert-annotated images.

Abstract

Otoscopy is a diagnostic procedure to examine the ear canal and eardrum using an otoscope. It identifies conditions like infections, foreign bodies, ear drum perforations and ear abnormalities. We propose a novel resource efficient deep learning and transformer based framework, Sumotosima (Summarizer for otoscopic images), an end-to-end pipeline for classification followed by summarization. Our framework works on combination of triplet and cross-entropy losses. Additionally, we use Knowledge Enhanced Multimodal BART whose input is fused textual and image embedding. The objective is to provide summaries that are well-suited for patients, ensuring clarity and efficiency in understanding otoscopic images. Given the lack of existing datasets, we have curated our own OCASD (Otoscopic Classification And Summary Dataset), which includes 500 images with 5 unique categories annotated with their class and summaries by Otolaryngologists. Sumotosima achieved a result of 98.03%, which is 7.00%, 3.10%, 3.01% higher than K-Nearest Neighbors, Random Forest and Support Vector Machines, respectively, in classification tasks. For summarization, Sumotosima outperformed GPT-4o and LLaVA by 88.53% and 107.57% in ROUGE scores, respectively. We have made our code and dataset publicly available at https://github.com/anas2908/Sumotosima