# Integrating clinical indications and patient demographics for multilabel abnormality classification and automated report generation in 3D chest CT scans

## TL;DR

This paper introduces a deep learning model that combines 3D chest CT scans with patient demographics and clinical data to improve abnormality detection and report generation.

## Contribution

A novel multimodal deep learning framework that integrates imaging and non-imaging data for enhanced CT scan analysis.

## Key findings

- Incorporating clinical and demographic data improves multilabel classification F1 score by 6.13%.
- Automated report generation benefits with F1 score increases of 14.78% and 6.69% for extended models.
- The approach enhances radiological workflows and abnormality detection accuracy.

## Abstract

The increasing number of computed tomography (CT) scan examinations and the time-intensive nature of manual analysis necessitate efficient automated methods to assist radiologists in managing their increasing workload. While deep learning approaches primarily classify abnormalities from three-dimensional (3D) CT images, radiologists also incorporate clinical indications and patient demographics, such as age and sex, for diagnosis. This study aims to enhance multilabel abnormality classification and automated report generation by integrating imaging and non-imaging data. We propose a multimodal deep learning model that combines 3D chest CT scans, clinical information reports, patient age, and sex to improve diagnostic accuracy. Our method extracts visual features from 3D volumes using a visual encoder, textual features from clinical indications via a pretrained language model, and demographic features through a lightweight feedforward neural network. These extracted features are projected into a shared representation space, concatenated, and processed by a projection head to predict abnormalities. For the multilabel classification task, incorporating clinical indications and patient demographics into an existing visual encoder, called CT-Net, improves the F1 score to 51.58, representing a +Δ6.13% increase over CT-Net alone. For the automated report generation task, we extend two existing methods, CT2Rep and CT-AGRG, by integrating clinical indications and demographic data. This integration enhances Clinical Efficacy metrics, yielding an F1 score improvement of +Δ14.78% for the CT2Rep extension and +Δ6.69% for the CT-AGRG extension. Our findings suggest that incorporating patient demographics and clinical information into deep learning frameworks can significantly improve automated CT scan analysis. This approach has the potential to enhance radiological workflows and facilitate more comprehensive and accurate abnormality detection in clinical practice.