Enhanced Transfer Learning Through Medical Imaging and Patient Demographic Data Fusion

TL;DR

This paper demonstrates that combining medical image features with patient demographic data significantly improves classification accuracy, especially when using transfer learning with pre-trained neural networks, with minimal additional computational cost.

Contribution

It introduces a method for enhancing medical image classification by fusing image features with patient metadata using transfer learning techniques.

Findings

Metadata improves classification performance.

Pre-trained models benefit from direct use or fine-tuning depending on image type.

Performance gains are achieved with negligible extra computation.

Abstract

In this work we examine the performance enhancement in classification of medical imaging data when image features are combined with associated non-image data. We compare the performance of eight state-of-the-art deep neural networks in classification tasks when using only image features, compared to when these are combined with patient metadata. We utilise transfer learning with networks pretrained on ImageNet used directly as feature extractors and fine tuned on the target domain. Our experiments show that performance can be significantly enhanced with the inclusion of metadata and use interpretability methods to identify which features lead to these enhancements. Furthermore, our results indicate that the performance enhancement for natural medical imaging (e.g. optical images) benefit most from direct use of pre-trained models, whereas non natural images (e.g. representations of non imaging data) benefit most from fine tuning pre-trained networks. These enhancements come at a negligible additional cost in computation time, and therefore is a practical method for other applications.