One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data

TL;DR

This paper introduces OvO attention, a scalable multimodal fusion method for biomedical data that reduces computational complexity from quadratic to linear, enabling efficient integration of multiple data modalities.

Contribution

The paper presents OvO attention, a novel domain-neutral mechanism that scales linearly with modalities, addressing the inefficiency of pairwise attention in high-modal scenarios.

Findings

OvO attention outperforms traditional fusion methods in accuracy.

It significantly reduces computational costs in multimodal data processing.

Demonstrated effectiveness on diverse biomedical datasets.

Abstract

Multimodal learning models have become increasingly important as they surpass single-modality approaches on diverse tasks ranging from question-answering to autonomous driving. Despite the importance of multimodal learning, existing efforts focus on NLP applications, where the number of modalities is typically less than four (audio, video, text, images). However, data inputs in other domains, such as the medical field, may include X-rays, PET scans, MRIs, genetic screening, clinical notes, and more, creating a need for both efficient and accurate information fusion. Many state-of-the-art models rely on pairwise cross-modal attention, which does not scale well for applications with more than three modalities. For $n$ modalities, computing attention will result in $n \choose 2$ operations, potentially requiring considerable amounts of computational resources. To address this, we propose a new domain-neutral attention mechanism, One-Versus-Others (OvO) attention, that scales linearly with the number of modalities and requires only $n$ attention operations, thus offering a significant reduction in computational complexity compared to existing cross-modal attention algorithms. Using three diverse real-world datasets as well as an additional simulation experiment, we show that our method improves performance compared to popular fusion techniques while decreasing computation costs.