Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records

TL;DR

This paper introduces PersAM, a Transformer-based personalized attention mechanism that adaptively identifies relevant regions in medical images by incorporating clinical records, improving diagnosis accuracy in pathology.

Contribution

The study presents a novel Transformer-based method that integrates clinical records with medical images to dynamically adjust attention regions for diagnosis.

Findings

Effective identification of attention regions in pathology images.

Improved accuracy in lymphoma subtype classification.

Demonstrated on large-scale digital pathology data.

Abstract

In medical image diagnosis, identifying the attention region, i.e., the region of interest for which the diagnosis is made, is an important task. Various methods have been developed to automatically identify target regions from given medical images. However, in actual medical practice, the diagnosis is made based not only on the images but also on a variety of clinical records. This means that pathologists examine medical images with some prior knowledge of the patients and that the attention regions may change depending on the clinical records. In this study, we propose a method called the Personalized Attention Mechanism (PersAM), by which the attention regions in medical images are adaptively changed according to the clinical records. The primary idea of the PersAM method is to encode the relationships between the medical images and clinical records using a variant of Transformer architecture. To demonstrate the effectiveness of the PersAM method, we applied it to a large-scale digital pathology problem of identifying the subtypes of 842 malignant lymphoma patients based on their gigapixel whole slide images and clinical records.