GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation

TL;DR

GTP-4o introduces a novel graph-based framework for multi-modal biomedical data that effectively handles missing modalities by embedding diverse clinical information into a unified, complete representation using modality prompts and graph aggregation.

Contribution

The paper proposes GTP-4o, a pioneering heterogeneous graph learning approach that completes missing modality embeddings and captures cross-modal relations for improved biomedical representation.

Findings

Outperforms prior state-of-the-art methods on benchmark datasets.

Effectively handles missing modalities with graph prompting mechanism.

Demonstrates robust cross-modal interaction and semantic property capture.

Abstract

Recent advances in learning multi-modal representation have witnessed the success in biomedical domains. While established techniques enable handling multi-modal information, the challenges are posed when extended to various clinical modalities and practical modalitymissing setting due to the inherent modality gaps. To tackle these, we propose an innovative Modality-prompted Heterogeneous Graph for Omnimodal Learning (GTP-4o), which embeds the numerous disparate clinical modalities into a unified representation, completes the deficient embedding of missing modality and reformulates the cross-modal learning with a graph-based aggregation. Specially, we establish a heterogeneous graph embedding to explicitly capture the diverse semantic properties on both the modality-specific features (nodes) and the cross-modal relations (edges). Then, we design a modality-prompted completion that enables completing the inadequate graph representation of missing modality through a graph prompting mechanism, which generates hallucination graphic topologies to steer the missing embedding towards the intact representation. Through the completed graph, we meticulously develop a knowledge-guided hierarchical cross-modal aggregation consisting of a global meta-path neighbouring to uncover the potential heterogeneous neighbors along the pathways driven by domain knowledge, and a local multi-relation aggregation module for the comprehensive cross-modal interaction across various heterogeneous relations. We assess the efficacy of our methodology on rigorous benchmarking experiments against prior state-of-the-arts. In a nutshell, GTP-4o presents an initial foray into the intriguing realm of embedding, relating and perceiving the heterogeneous patterns from various clinical modalities holistically via a graph theory. Project page: https://gtp-4-o.github.io/.