MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging

TL;DR

MIRNet is a novel framework that combines self-supervised learning, graph-based reasoning, and clinical constraints to improve diagnostic accuracy in medical imaging, demonstrated on tongue diagnosis with a new large dataset.

Contribution

The paper introduces MIRNet, integrating pre-training, graph reasoning, and clinical priors for medical image diagnosis, along with a new tongue image dataset, TongueAtlas-4K.

Findings

Achieves state-of-the-art performance on tongue diagnosis

Effectively models label correlations with graph attention networks

Utilizes a large, expert-curated tongue image dataset

Abstract

Automated interpretation of medical images demands robust modeling of complex visual-semantic relationships while addressing annotation scarcity, label imbalance, and clinical plausibility constraints. We introduce MIRNet (Medical Image Reasoner Network), a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning. Tongue image diagnosis is a particularly challenging domain that requires fine-grained visual and semantic understanding. Our approach leverages self-supervised masked autoencoder (MAE) to learn transferable visual representations from unlabeled data; employs graph attention networks (GAT) to model label correlations through expert-defined structured graphs; enforces clinical priors via constraint-aware optimization using KL divergence and regularization losses; and mitigates imbalance using asymmetric loss (ASL) and boosting ensembles. To address annotation scarcity, we also introduce TongueAtlas-4K, a comprehensive expert-curated benchmark comprising 4,000 images annotated with 22 diagnostic labels--representing the largest public dataset in tongue analysis. Validation shows our method achieves state-of-the-art performance. While optimized for tongue diagnosis, the framework readily generalizes to broader diagnostic medical imaging tasks.