Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform

TL;DR

This paper introduces DAFT, a novel CNN module that dynamically integrates 3D medical images with tabular clinical data, significantly improving Alzheimer's diagnosis and prognosis accuracy.

Contribution

The paper presents DAFT, a new module for CNNs that effectively combines image and tabular data, enhancing diagnostic and predictive performance in medical applications.

Findings

DAFT outperforms existing CNNs in Alzheimer's diagnosis.

DAFT achieves a mean balanced accuracy of 0.622.

DAFT attains a mean c-index of 0.748 for dementia prediction.

Abstract

Prior work on diagnosing Alzheimer's disease from magnetic resonance images of the brain established that convolutional neural networks (CNNs) can leverage the high-dimensional image information for classifying patients. However, little research focused on how these models can utilize the usually low-dimensional tabular information, such as patient demographics or laboratory measurements. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that dynamically rescales and shifts the feature maps of a convolutional layer, conditional on a patient's tabular clinical information. We show that DAFT is highly effective in combining 3D image and tabular information for diagnosis and time-to-dementia prediction, where it outperforms competing CNNs with a mean balanced accuracy of 0.622 and mean c-index of 0.748, respectively. Our extensive ablation study provides valuable insights into the architectural properties of DAFT. Our implementation is available at https://github.com/ai-med/DAFT.