Xray2Xray: World Model from Chest X-rays with Volumetric Context

TL;DR

Xray2Xray introduces a world model that learns 3D structural representations from 2D chest X-rays, improving disease diagnosis and risk prediction by capturing volumetric context.

Contribution

It presents a novel approach to encode 3D information from 2D X-rays using a latent transition model, enhancing diagnostic accuracy.

Findings

Outperforms supervised and self-supervised methods in risk estimation

Achieves competitive accuracy in pathology classification

Can reconstruct volumetric context from latent representations

Abstract

Chest X-rays (CXRs) are the most widely used medical imaging modality and play a pivotal role in diagnosing diseases. However, as 2D projection images, CXRs are limited by structural superposition, which constrains their effectiveness in precise disease diagnosis and risk prediction. To address the limitations of 2D CXRs, this study introduces Xray2Xray, a novel World Model that learns latent representations encoding 3D structural information from chest X-rays. Xray2Xray captures the latent representations of the chest volume by modeling the transition dynamics of X-ray projections across different angular positions with a vision model and a transition model. We employed the latent representations of Xray2Xray for downstream risk prediction and disease diagnosis tasks. Experimental results showed that Xray2Xray outperformed both supervised methods and self-supervised pretraining methods for cardiovascular disease risk estimation and achieved competitive performance in classifying five pathologies in CXRs. We also assessed the quality of Xray2Xray's latent representations through synthesis tasks and demonstrated that the latent representations can be used to reconstruct volumetric context.