Hypernetwork-Based Adaptive Aggregation for Multimodal Multiple-Instance Learning in Predicting Coronary Calcium Debulking

TL;DR

This paper introduces HyperAdAgFormer, a hypernetwork-based transformer model that adaptively aggregates features for predicting the necessity of coronary calcium debulking from CT images, addressing the challenge of variable decision criteria among physicians.

Contribution

The paper presents the first application of a hypernetwork-based adaptive aggregation transformer for multimodal MIL in coronary calcium debulking prediction, incorporating patient-specific data for improved accuracy.

Findings

HyperAdAgFormer outperforms baseline models on clinical data.

Adaptive aggregation improves prediction accuracy.

Code is publicly available for reproducibility.

Abstract

In this paper, we present the first attempt to estimate the necessity of debulking coronary artery calcifications from computed tomography (CT) images. We formulate this task as a Multiple-instance Learning (MIL) problem. The difficulty of this task lies in that physicians adjust their focus and decision criteria for device usage according to tabular data representing each patient's condition. To address this issue, we propose a hypernetwork-based adaptive aggregation transformer (HyperAdAgFormer), which adaptively modifies the feature aggregation strategy for each patient based on tabular data through a hypernetwork. The experiments using the clinical dataset demonstrated the effectiveness of HyperAdAgFormer. The code is publicly available at https://github.com/Shiku-Kaito/HyperAdAgFormer.