VVGT: Visual Volume-Grounded Transformer

TL;DR

VVGT introduces a fast, scalable, and accurate volumetric visualization method using a transformer-based framework that directly maps data to 3D Gaussian splatting, surpassing traditional DVR and existing 3DGS approaches.

Contribution

The paper proposes VVGT, a novel feed-forward, volumetric representation framework employing a dual-transformer and epipolar attention, eliminating the need for per-scene optimization and enabling real-time visualization.

Findings

Achieves high-quality visualization with faster conversion times.

Demonstrates improved geometric consistency and zero-shot generalization.

Enables interactive and scalable volumetric visualization across diverse datasets.

Abstract

Volumetric visualization has long been dominated by Direct Volume Rendering (DVR), which operates on dense voxel grids and suffers from limited scalability as resolution and interactivity demands increase. Recent advances in 3D Gaussian Splatting (3DGS) offer a representation-centric alternative; however, existing volumetric extensions still depend on costly per-scene optimization, limiting scalability and interactivity. We present VVGT (Visual Volume-Grounded Transformer), a feed-forward, representation-first framework that directly maps volumetric data to a 3D Gaussian Splatting representation, advancing a new paradigm for volumetric visualization beyond DVR. Unlike prior feed-forward 3DGS methods designed for surface-centric reconstruction, VVGT explicitly accounts for volumetric rendering, where each pixel aggregates contributions along a ray. VVGT employs a dual-transformer network and introduces Volume Geometry Forcing, an epipolar cross-attention mechanism that integrates multi-view observations into distributed 3D Gaussian primitives without surface assumptions. This design eliminates per-scene optimization while enabling accurate volumetric representations. Extensive experiments show that VVGT achieves high-quality visualization with orders-of-magnitude faster conversion, improved geometric consistency, and strong zero-shot generalization across diverse datasets, enabling truly interactive and scalable volumetric visualization. The code will be publicly released upon acceptance.