Accelerating 3D Gaussian Splatting with Neural Sorting and Axis-Oriented Rasterization

TL;DR

This paper introduces a co-designed architecture-algorithm approach for 3D Gaussian Splatting that significantly accelerates rendering and reduces energy consumption on resource-limited devices by optimizing rasterization and sorting processes.

Contribution

It proposes axis-oriented rasterization and neural sorting techniques, along with a reconfigurable processing array, to enhance efficiency and speed in 3D Gaussian Splatting.

Findings

Achieves 23.4 to 27.8 times speedup over edge GPUs.

Reduces multiply-and-add operations by up to 63%.

Decreases energy consumption by 28.8 to 51.4 times.

Abstract

3D Gaussian Splatting (3DGS) has recently gained significant attention for high-quality and efficient view synthesis, making it widely adopted in fields such as AR/VR, robotics, and autonomous driving. Despite its impressive algorithmic performance, real-time rendering on resource-constrained devices remains a major challenge due to tight power and area budgets. This paper presents an architecture-algorithm co-design to address these inefficiencies. First, we reveal substantial redundancy caused by repeated computation of common terms/expressions during the conventional rasterization. To resolve this, we propose axis-oriented rasterization, which pre-computes and reuses shared terms along both the X and Y axes through a dedicated hardware design, effectively reducing multiply-and-add (MAC) operations by up to 63%. Second, by identifying the resource and performance inefficiency of the sorting process, we introduce a novel neural sorting approach that predicts order-independent blending weights using an efficient neural network, eliminating the need for costly hardware sorters. A dedicated training framework is also proposed to improve its algorithmic stability. Third, to uniformly support rasterization and neural network inference, we design an efficient reconfigurable processing array that maximizes hardware utilization and throughput. Furthermore, we introduce a $\pi$-trajectory tile schedule, inspired by Morton encoding and Hilbert curve, to optimize Gaussian reuse and reduce memory access overhead. Comprehensive experiments demonstrate that the proposed design preserves rendering quality while achieving a speedup of $23.4\sim27.8\times$ and energy savings of $28.8\sim51.4\times$ compared to edge GPUs for real-world scenes. We plan to open-source our design to foster further development in this field.