Revisiting Integration of Image and Metadata for DICOM Series Classification: Cross-Attention and Dictionary Learning

TL;DR

This paper introduces a multimodal framework combining image content and metadata with cross-attention and dictionary learning for robust DICOM series classification, addressing metadata inconsistencies and missing data.

Contribution

It presents a novel end-to-end approach that explicitly models metadata sparsity and cross-modal interactions without imputation, improving classification robustness.

Findings

Outperforms image-only, metadata-only, and baseline multimodal methods.

Handles missing and inconsistent metadata without imputation.

Demonstrates improved robustness and generalization across datasets.

Abstract

Automated identification of DICOM image series is essential for large-scale medical image analysis, quality control, protocol harmonization, and reliable downstream processing. However, DICOM series classification remains challenging due to heterogeneous slice content, variable series length, and entirely missing, incomplete or inconsistent DICOM metadata. We propose an end-to-end multimodal framework for DICOM series classification that jointly models image content and acquisition metadata while explicitly accounting for all these challenges. (i) Images and metadata are encoded with modality-aware modules and fused using a bi-directional cross-modal attention mechanism. (ii) Metadata is processed by a sparse, missingness-aware encoder based on learnable feature dictionaries and value-conditioned modulation. By design, the approach does not require any form of imputation. (iii) Variability in series length and image data dimensions is handled via a 2.5D visual encoder and attention operating on equidistantly sampled slices. We evaluate the proposed approach on the publicly available Duke Liver MRI dataset and a large multi-institutional in-house cohort, assessing both in-domain performance and out-of-domain generalization. Across all evaluation settings, the proposed method consistently outperforms relevant image only, metadata-only and multimodal 2D/3D baselines. The results demonstrate that explicitly modeling metadata sparsity and cross-modal interactions improves robustness for DICOM series classification.