The LUMirage: An independent evaluation of zero-shot performance in the LUMIR challenge
Rohit Jena, Pratik Chaudhari, James C. Gee

TL;DR
This paper critically re-evaluates the zero-shot performance claims of deep learning methods in neuroimaging registration, revealing limitations on out-of-distribution data, scalability issues, and sensitivity to preprocessing, thus emphasizing the need for realistic evaluation protocols.
Contribution
It provides an independent, rigorous assessment of zero-shot generalization in neuroimaging registration, challenging optimistic claims and highlighting practical limitations of deep learning methods.
Findings
Deep learning methods perform well on in-distribution T1w images and macaque data.
Performance drops significantly on out-of-distribution contrasts like T2 and FLAIR.
Deep learning methods struggle with high-resolution images and are sensitive to preprocessing choices.
Abstract
The LUMIR challenge represents an important benchmark for evaluating deformable image registration methods on large-scale neuroimaging data. While the challenge demonstrates that modern deep learning methods achieve competitive accuracy on T1-weighted MRI, it also claims exceptional zero-shot generalization to unseen contrasts and resolutions, assertions that contradict established understanding of domain shift in deep learning. In this paper, we perform an independent re-evaluation of these zero-shot claims using rigorous evaluation protocols while addressing potential sources of instrumentation bias. Our findings reveal a more nuanced picture: (1) deep learning methods perform comparably to iterative optimization on in-distribution T1w images and even on human-adjacent species (macaque), demonstrating improved task understanding; (2) however, performance degrades significantly on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced MRI Techniques and Applications · Advanced Neuroimaging Techniques and Applications · Medical Imaging Techniques and Applications
