Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncology
Omnia Alwazzan, Amaya Gallagher-Syed, Thomas O. Millner, Sebastian, Brandner, Ioannis Patras, Silvia Marino, Gregory Slabaugh

TL;DR
This paper introduces a dual fusion approach integrating omic data with whole slide images at both local and global levels, improving CNS tumor classification and survival prediction in precision oncology.
Contribution
It proposes a novel multimodal fusion framework using early and late fusion with omic embeddings and MOAB, enhancing interpretability and diagnostic accuracy.
Findings
Achieved accurate CNS tumor subtyping across 20 subtypes.
Improved survival prediction on TCGA-BLCA dataset.
Demonstrated competitive performance on TCGA-BRCA dataset.
Abstract
The integration of DNA methylation data with a Whole Slide Image (WSI) offers significant potential for enhancing the diagnostic precision of central nervous system (CNS) tumor classification in neuropathology. While existing approaches typically integrate encoded omic data with histology at either an early or late fusion stage, the potential of reintroducing omic data through dual fusion remains unexplored. In this paper, we propose the use of omic embeddings during early and late fusion to capture complementary information from local (patch-level) to global (slide-level) interactions, boosting performance through multimodal integration. In the early fusion stage, omic embeddings are projected onto WSI patches in latent-space, which generates embeddings that encapsulate per-patch molecular and morphological insights. This effectively incorporates omic information into the spatial…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBrain Tumor Detection and Classification · Advanced Image Fusion Techniques
MethodsSoftmax · Attention Is All You Need
