Graph Memory: A Structured and Interpretable Framework for Modality-Agnostic Embedding-Based Inference

TL;DR

Graph Memory (GM) is a novel, interpretable, and modality-agnostic framework that models embedding spaces with a graph of prototypes, enabling effective, transparent inference across diverse data types.

Contribution

The paper introduces Graph Memory, a structured non-parametric model that unifies instance retrieval, reasoning, and diffusion, with modality-agnostic capabilities and improved calibration.

Findings

Matches or exceeds kNN and Label Spreading accuracy

Provides better calibration and smoother decision boundaries

Uses significantly less memory than comparable methods

Abstract

We introduce Graph Memory (GM), a structured non-parametric framework that represents an embedding space through a compact graph of reliability-annotated prototype regions. GM encodes local geometry and regional ambiguity through prototype relations and performs inference by diffusing query evidence across this structure, unifying instance retrieval, prototype-based reasoning, and graph diffusion within a single inductive and interpretable model. The framework is inherently modality-agnostic: in multimodal settings, independent prototype graphs are constructed for each modality and their calibrated predictions are combined through reliability-aware late fusion, enabling transparent integration of heterogeneous sources such as whole-slide images and gene-expression profiles. Experiments on synthetic benchmarks, breast histopathology (IDC), and the multimodal AURORA dataset show that GM matches or exceeds the accuracy of kNN and Label Spreading while providing substantially better calibration, smoother decision boundaries, and an order-of-magnitude smaller memory footprint. By explicitly modeling regional reliability and relational structure, GM offers a principled and interpretable approach to non-parametric inference across single- and multi-modal domains.