Single-Slice-to-3D Reconstruction in Medical Imaging and Natural Objects: A Comparative Benchmark with SAM 3D

TL;DR

This paper benchmarks five state-of-the-art models for single-slice 3D reconstruction in medical imaging, revealing fundamental limitations and highlighting the need for domain-specific adaptation to improve accuracy.

Contribution

It provides a comprehensive comparison of existing models on medical and natural datasets, quantifies their limitations, and emphasizes the importance of domain-specific constraints for reliable 3D reconstruction.

Findings

Voxel-based overlap remains low across models due to depth ambiguity.

SAM3D best captures topological similarity despite volumetric failure.

Reliable 3D medical reconstruction requires domain-specific adaptation.

Abstract

While three-dimensional imaging is essential for clinical diagnosis, its high cost and long wait times have motivated the use of image-to-3D foundation models to infer volume from two-dimensional modalities. However, because these models are trained on natural images, their learned geometric priors struggle to transfer to inherently planar medical data. A benchmark of five state-of-the-art models (SAM3D, Hunyuan3D-2.1, Direct3D, Hi3DGen, and TripoSG) across six medical and two natural datasets revealed that voxel-based overlap remains uniformly low across all methods due to severe depth ambiguity from single-slice inputs. Despite this fundamental volumetric failure, global distance metrics indicate that SAM3D best captures topological similarity to ground-truth medical shapes, whereas alternative models are prone to oversimplification. Ultimately, these findings quantify the limits of zero-shot single-slice 3D inference, highlighting that reliable medical 3D reconstruction requires domain-specific adaptation and anatomical constraints to overcome complex medical geometries.