Why Relational Graphs Will Save the Next Generation of Vision Foundation Models?

TL;DR

This paper argues that integrating explicit relational graphs into vision foundation models enhances their reasoning, robustness, and efficiency, especially for tasks requiring understanding of entities, roles, and relations.

Contribution

It proposes the use of dynamic relational graphs as an explicit interface in vision models, demonstrating improved performance and interpretability in various vision tasks.

Findings

Augmenting FMs with graph modules improves semantic fidelity.

Graph hybrids enhance out-of-distribution robustness.

They offer better memory and hardware efficiency.

Abstract

Vision foundation models (FMs) have become the predominant architecture in computer vision, providing highly transferable representations learned from large-scale, multimodal corpora. Nonetheless, they exhibit persistent limitations on tasks that require explicit reasoning over entities, roles, and spatio-temporal relations. Such relational competence is indispensable for fine-grained human activity recognition, egocentric video understanding, and multimodal medical image analysis, where spatial, temporal, and semantic dependencies are decisive for performance. We advance the position that next-generation FMs should incorporate explicit relational interfaces, instantiated as dynamic relational graphs (graphs whose topology and edge semantics are inferred from the input and task context). We illustrate this position with cross-domain evidence from recent systems in human manipulation action recognition and brain tumor segmentation, showing that augmenting FMs with lightweight, context-adaptive graph-reasoning modules improves fine-grained semantic fidelity, out of distribution robustness, interpretability, and computational efficiency relative to FM only baselines. Importantly, by reasoning sparsely over semantic nodes, such hybrids also achieve favorable memory and hardware efficiency, enabling deployment under practical resource constraints. We conclude with a targeted research agenda for FM graph hybrids, prioritizing learned dynamic graph construction, multi-level relational reasoning (e.g., part object scene in activity understanding, or region organ in medical imaging), cross-modal fusion, and evaluation protocols that directly probe relational competence in structured vision tasks.