Render-FM: A Foundation Model for Real-time Photorealistic Volumetric Rendering

TL;DR

Render-FM introduces a foundation model that enables real-time, high-quality volumetric rendering of CT scans without per-scan optimization, significantly improving efficiency and clinical applicability.

Contribution

It presents a novel foundation model using large-scale pre-training to achieve real-time, high-fidelity volumetric rendering of CT scans without scene-specific optimization.

Findings

Achieves visual fidelity comparable or superior to specialized methods.

Reduces rendering preparation time from nearly an hour to seconds.

Enables real-time interactive 3D visualization for clinical workflows.

Abstract

Volumetric rendering of Computed Tomography (CT) scans is crucial for visualizing complex 3D anatomical structures in medical imaging. Current high-fidelity approaches, especially neural rendering techniques, require time-consuming per-scene optimization, limiting clinical applicability due to computational demands and poor generalizability. We propose Render-FM, a novel foundation model for direct, real-time volumetric rendering of CT scans. Render-FM employs an encoder-decoder architecture that directly regresses 6D Gaussian Splatting (6DGS) parameters from CT volumes, eliminating per-scan optimization through large-scale pre-training on diverse medical data. By integrating robust feature extraction with the expressive power of 6DGS, our approach efficiently generates high-quality, real-time interactive 3D visualizations across diverse clinical CT data. Experiments demonstrate that Render-FM achieves visual fidelity comparable or superior to specialized per-scan methods while drastically reducing preparation time from nearly an hour to seconds for a single inference step. This advancement enables seamless integration into real-time surgical planning and diagnostic workflows. The project page is: https://gaozhongpai.github.io/renderfm/.