Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification

TL;DR

This paper introduces a multi-modal learning framework that integrates ophthalmologists' diagnostic insights with OCT imaging data, enhancing disease classification accuracy and interpretability.

Contribution

It presents a novel dual-stream architecture that combines expert diagnostic attributes with visual features, outperforming existing methods in OCT classification.

Findings

Diagnostic attribute features improve classification accuracy.

The dual-stream model surpasses state-of-the-art approaches.

Analysis identifies key components contributing to performance.

Abstract

In this paper, we propose a framework that incorporates experts diagnostics and insights into the analysis of Optical Coherence Tomography (OCT) using multi-modal learning. To demonstrate the effectiveness of this approach, we create a medical diagnostic attribute dataset to improve disease classification using OCT. Although there have been successful attempts to deploy machine learning for disease classification in OCT, such methodologies lack the experts insights. We argue that injecting ophthalmological assessments as another supervision in a learning framework is of great importance for the machine learning process to perform accurate and interpretable classification. We demonstrate the proposed framework through comprehensive experiments that compare the effectiveness of combining diagnostic attribute features with latent visual representations and show that they surpass the state-of-the-art approach. Finally, we analyze the proposed dual-stream architecture and provide an insight that determine the components that contribute most to classification performance.