Scale-GS: Efficient Scalable Gaussian Splatting via Redundancy-filtering Training on Streaming Content

TL;DR

This paper introduces Scale-GS, a scalable Gaussian Splatting framework that improves training efficiency and rendering quality for dynamic scenes by hierarchical organization, selective activation, motion modeling, and adaptive masking.

Contribution

It proposes a hierarchical, anchor-based organization of Gaussians, a hybrid motion modeling strategy, and an adaptive masking mechanism to enhance efficiency in streaming 3D Gaussian Splatting.

Findings

Achieves higher visual quality with less training time.

Reduces computational overhead significantly.

Effective in dynamic scene rendering.

Abstract

3D Gaussian Splatting (3DGS) enables high-fidelity real-time rendering, a key requirement for immersive applications. However, the extension of 3DGS to dynamic scenes remains limitations on the substantial data volume of dense Gaussians and the prolonged training time required for each frame. This paper presents \M, a scalable Gaussian Splatting framework designed for efficient training in streaming tasks. Specifically, Gaussian spheres are hierarchically organized by scale within an anchor-based structure. Coarser-level Gaussians represent the low-resolution structure of the scene, while finer-level Gaussians, responsible for detailed high-fidelity rendering, are selectively activated by the coarser-level Gaussians. To further reduce computational overhead, we introduce a hybrid deformation and spawning strategy that models motion of inter-frame through Gaussian deformation and triggers Gaussian spawning to characterize wide-range motion. Additionally, a bidirectional adaptive masking mechanism enhances training efficiency by removing static regions and prioritizing informative viewpoints. Extensive experiments demonstrate that \M~ achieves superior visual quality while significantly reducing training time compared to state-of-the-art methods.

Tables

Table 1. TABLE I: The quantitative results on different datasets. NV, MR, and GI are separately denoted NV3D dataset, Meeting room dataset, and Google Immersive dataset. The best and second-best results are red and purple , respectively.

	SSIM↑			PSNR↑			LPIPS↓			Training time(s)↓			FPS↑
Method	NV	MR	GI	NV	MR	GI	NV	MR	GI	NV	MR	GI	NV	MR	GI
Deform	0.956	0.857	0.843	32.10	27.81	26.46	0.127	0.206	0.216	38	29	81	40	45	35
4DGS	0.959	0.870	0.853	32.23	28.69	27.88	0.109	0.184	0.214	8.2	7.5	73.5	30	38	26
3DG-S	0.958	0.906	0.868	32.93	29.09	28.60	0.101	0.145	0.209	9.6	7.7	76.1	210	252	190
HiCoM	0.967	0.909	0.852	33.28	29.15	28.68	0.111	0.152	0.208	7.1	3.9	60.8	256	284	212
IGS	0.965	0.909	0.909	33.62	30.13	29.72	0.109	0.143	0.168	3.6	3.2	43.2	204	251	186
Scale-GS	0.966	0.908	0.912	34.47	31.58	31.18	0.106	0.142	0.169	3.2	3.0	37.3	274	276	199

Table 2. TABLE II: Ablation study of scale number on Google Immersive Dataset, where time indicates training time.

Scale	SSIM↑	PSNR↑	LPIPS↓	Time(s)↓
2	0.893	30.602	0.168	36
3	0.916	31.233	0.154	33
4	0.881	29.917	0.170	35
5	0.857	28.242	0.191	39

Table 3. TABLE III: Ablation study of the hybrid deformation-spawning policy, where time indicates training time.

	SSIM↑	PSNR↑	LPIPS↓	Time(s)↓
deform + spawn	0.913	31.602	0.138	2.9
w/o deformation	0.854	26.719	0.245	4.6
w/o spawning	0.887	29.304	0.127	3.8

Table 4. TABLE IV: The ablation study of viewpoint selection results, where time indicates training time.

	BAM selection		w/o BAM selection
Scene	PSNR↑	SSIM↑	PSNR↑	SSIM↑
Car	31.320${}_{\text{+1.012}}$	0.917${}_{\text{+0.011}}$	30.308	0.906
Goats	31.778${}_{\text{+1.150}}$	0.922${}_{\text{+0.007}}$	30.628	0.915
Dogs	31.022${}_{\text{+0.737}}$	0.914${}_{\text{+0.021}}$	30.285	0.893
Paint	31.751${}_{\text{+0.906}}$	0.916${}_{\text{+0.006}}$	30.845	0.910
Welder	30.281${}_{\text{+1.729}}$	0.901${}_{\text{+0.035}}$	28.552	0.866

Equations

$$G_i(x)=\exp \left(-\frac{1}{2}(x-{\mu }_i{)}^{\top }{\Sigma }_i^{-1}(x-{\mu }_i)\right),$$

$$C(x^{\prime })=\sum _{i\in N}{c}_i{\alpha }_i\prod _{j=1}^{i-1}(1-{\alpha }_j),$$

$$\{{\mu }_0,\dots ,{\mu }_{k-1}\}=x_v+\{O_0,\dots ,O_{k-1}\}\cdot c_v.$$

$$\{{\alpha }_0,\dots ,{\alpha }_{k-1}\}=F_{\alpha }({\hat{f}}_v,{\delta }_v,{\vec{d}}_v).$$

$$s_i=\text{clamp}(s_{\text{min}}^{(l)},s_{\text{max}}^{(l)},s_i),\text{if }{s}_i\in l.$$

$$\Delta {\mu }_i^{(l)},\Delta q_i^{(l)}=\mathrm{ML}{\mathrm{P}}_{\mathrm{g}}(h({\mu }_{i,t-1}^{(l)})).$$

$$c_{i,t}^{(l)},{\alpha }_{i,t}^{(l)}=\mathrm{ML}{\mathrm{P}}_{\mathrm{a}}(c_{i,t-1}^{(l)},{\alpha }_{i,t-1}^{(l)}).$$

$${\theta }_i^{(l,t)}$$

$$q_{i,t}^{(l)}$$

$$C^{(l)}(x^{\prime })=\sum _{i\in N}{c}_{i,t}^{(l)}{\alpha }_{i,t}^{(l)}\prod _{j=1}^{i-1}(1-{\alpha }_{j,t}^{(l)}).$$

$${\mathcal{L}}^{(l)}={\mathcal{L}}_1^{(l)}(I_{n,t}^l,{\hat{C}}_{n,t}^l)+{\lambda }_{\text{SSIM}}{\mathcal{L}}_{\text{SSIM}}^{(l)}(I_{n,t}^l,{\hat{C}}_{n,t}^l),$$

$$\nabla g_v^{(l)}=\frac{1}{|G_v^l|}\sum _{i\in G_v^l}\Vert \frac{\partial \mathcal{L}}{\partial {\theta }_i^{(l)}}\Vert ,$$

$${\tau }_{\text{add}}^{(l)}=\frac{Vol}{4^{l-1}},$$

$$C(x^{\prime })=\sum _{i\in N}\sigma (M_{v,i})c_{i,t}{\alpha }_{i,t}\prod _{j=1}^{i-1}(1-\sigma (M_{v,i}){\alpha }_{j,t}).$$

$$\mathcal{L}=\sum _{l=1}^L{\mathcal{L}}^{(l)}+{\lambda }_{\text{r}}\sum _{v\in V}\sum _{i=1}^N\sigma (M_{v,i}),$$

$$S(c_k)=\sum _{v\in V}1\left[\mathrm{IoU}(c_k,v)>{\tau }_{\text{view}}\right]\cdot \omega (c_k,v),$$

$$\omega (c_k,v)=|{\mathbf{n}}_v^{\top }{\mathbf{d}}_{c_k}|,$$

Full text

Keywords: Streaming Gaussian Splatting, multi-scale representation, dynamic scene rendering, novel view synthesis.

I Introduction

The rapid advancement of 3D Gaussian Splatting (3DGS) [1] has significantly reshaped the domain of real-time 3D rendering. In particular, the introduction of Gaussian training methods designed for dynamic scenes [2, 3, 4, 5, 6, 7, 8, 9] has greatly enhanced the feasibility of 3D streaming applications, including virtual reality (VR), augmented reality (AR), and immersive telepresence systems. By explicitly representing scenes with differentiable Gaussians, these methods facilitate real-time rendering—capabilities that are critical for interactive applications demanding low-latency visual feedback. However, as computational demands scale sharply with scene and temporal complexity, the training time for dynamic scenes—ranging from tens of minutes to hours—conflicts with the low-latency requirements of real-time streaming.

The primary reason for the slow training time in Gaussian Splatting stems from redundant computations involving Gaussian spheres. The standard 3D Gaussian splatting methods process each frame independently, thereby incurring repetitive calculations on predominantly static Gaussians due to the limited extent of dynamic regions. Although recently some research partitions the scene into static and dynamic components to focus computational resources on dynamic Gaussians, the overall volume remains substantial. Consequently, significant overlapping computations occur across spatially and temporally adjacent regions, leading to inefficient resource usage and bottlenecks that hinder real-time performance.

We observe that the size and number of Gaussian spheres to represent the 3D scene vary significantly depending on scene complexity. For example, textureless planar regions can be effectively modeled by a small number of large Gaussian spheres, whereas highly textured areas necessitate dozens or even hundreds of smaller Gaussians. The larger Gaussian spheres often contribute more significantly to scene representation and therefore warrant higher training priority. Inspired from scalable video coding, this work proposes Scale-GS, a scalable Gaussian splatting framework aimed at mitigating redundancy in 3D spatial representation, thereby enabling accelerated training. Specifically, Gaussian spheres are organized by scale, with each level independently trained under the viewpoints at corresponding resolutions. For each frame, large scale Gaussians are first optimized using low-resolution views to approximate the scene. Upon convergence, a triggering criterion evaluates whether small scale Gaussians should be activated for refinement, thereby reducing the redundant Gaussian spheres optimization. As shown in Fig. 1, the proposed method achieves high rendering quality with short training time compared to the SOTA algorithm. The framework conducts a coarse-to-fine principle where the training at each level of scale is triggered by the preceding level of scale, thereby filtering redundant training of irrelevant Gaussians.

To enable efficient training for streaming content, we introduce a hybrid strategy that combines deformation and spawning to infer dynamic changes in the current frame based on the preceding frame. Generally, the Gaussian deformation [9, 6] models the motion of Gaussian sphere, but insufficient for capturing newly appearing objects or wide-range motion. In contrast, Gaussian spawning [10, 11] introduces new Gaussians to fine-tune dynamic regions but requires considerably longer training time. To balance the trade-off, the hybrid strategy first applies deformation to model inter-frame motion and then determines, based on the training outcome, whether spawning should be triggered for finer-grained refinement. Specifically, under the Scale-GS framework, if the deformation of a determined scale fails to adequately represent the dynamic, either the emergence of new content or the motion of smaller-scale Gaussians happens. Thus, the deformation result triggers new Gaussians spawning at the current scale and deformation at the next scale. This sequential activation ensures Gaussians are progressively introduced at locations with actual dynamics along with increasingly finer scales, enabling efficient and high-fidelity temporal GS representation.

To enhance training efficiency, we propose a bidirectional adaptive masking mechanism that simultaneously suppresses static regions and selects informative training viewpoints. The forward masking component detects dynamic and static anchors via inter-frame change analysis, where pixel-wise differences between consecutive frames are back-projected to estimate motion patterns. For backward camera viewpoint selection, we define a relevance score between projected dynamic anchors and camera fields of view, further weighted by directional factors that prioritize orthogonal or novel viewpoints. The top-ranked views, as determined by the relevance score, are selected to form the active viewpoint set. This bidirectional masking mechanism reduces computational redundancy caused by uninformative viewpoints and facilitates accurate reconstruction of dynamic scenes.

Comprehensive experiments conducted on three challenging real-world datasets—NV3D, MeetRoom, and Google Immersive—demonstrate the superior performance of the proposed framework across multiple evaluation metrics. Qualitative comparisons show that our method reconstructs significantly sharper fine-grained details, particularly in complex scenarios involving human interactions, dynamic phenomena such as flames, and intricate textures. Furthermore, experimental results demonstrate that Scale-GS not only improves visual quality but also outperforms current state-of-the-art methods in both training and rendering time. These findings validate the effectiveness of Scale-GS, which prioritizes more important Gaussian spheres and improves the average training efficiency.

The main contributions of this work are as follows:

We propose Scale-GS, a scalable GS framework performing redundancy-filtering training on streaming content to improve the efficiency. The  Scale-GS achieves the most efficient training compared to the existing Gaussian training methods on streaming content. 2. 2.

The Scale-GS integrates a hybrid deformation-spawning Gaussian training strategy that prioritizes large-scale Gaussians and selectively activating finer ones to reduce redundancy in 3D scene representation while preserving high-fidelity dynamic representations. 3. 3.

Extensive evaluations show that Scale-GS achieves superior efficiency–quality trade-offs for streaming novel view synthesis, improving visual quality, reducing training time, and supporting real-time rendering.

In the following, we first introduce the research related to the Scale-GS, including the novel view synthesis for static scene and videography at Sec. II. Later we thoroughly present the detail of the method at Sec. III. The qualitative and quantitative experimental results are exhibits at Sec. IV. Finally, we draw the conclusion and propose the future work at Sec. V.

II Related work

In this section, we separately review research on implicit and explicit novel view synthesis methods for both static and dynamic scenes. These studies primarily focus on improving visual quality and enhancing training efficiency.

II-A Novel View Synthesis for Static Scenes

Early novel view synthesis methods predominantly rely on geometric interpolation, with approaches such as the Lumigraph [12] and Light Field rendering [13, 14], laying the groundwork through advanced interpolation techniques applied to densely sampled input images.

Neural Radiance Fields (NeRF) [15] introduces a breakthrough in photorealistic view synthesis by modeling scene radiance through implicit neural representations using multi-layer perceptrons. This innovation has spurred extensive research aiming at overcoming NeRF’s inherent limitations across various dimensions. Key efforts include accelerating training procedures [16, 17, 18, 19], achieving real-time rendering performance [20, 21, 22], improving synthesis fidelity in complex scenes [23, 24, 21], and enhancing robustness under sparse input conditions [25, 26, 27]. However, the computational overhead inherent in NeRF’s volume rendering paradigm—which requires numerous neural network computation per frame—presents significant challenges in balancing training efficiency, rendering speed, and visual fidelity.

To address these limitations, Kerbl et al. [1] proposes 3DGS, which leverages explicit 3D Gaussian primitives combined with differentiable rasterization-based rendering to enable real-time, high-quality view synthesis. The 3DGS inspires a broad range of research efforts exploring various aspects of Gaussian-based scene representations. Some studies focus on enhancing rendering fidelity [28, 29, 30], while others aim to improve geometric precision and accuracy [31, 32]. Furthermore, considerable efforts are devoted to developing compression techniques to mitigate storage overhead [33, 28, 34, 35, 36]. Recent studies investigate the joint optimization of camera parameters alongside Gaussian field estimation [37], as well as the extension of Gaussian splatting to broader 3D content generation tasks [38, 39, 40]. Despite the 3DGS in static scenes achieves high-quality rendering, the development of an on-demand training framework for 3DGS remains an open challenge.

Generalization is introduced as an effective strategy to enhance inference speed. Recent developments in NeRF methodologies [41, 42, 27] and 3DGS[43, 44, 45] focus on generalizable reconstruction networks trained on large-scale datasets. Specifically, PixelSplat [46] uses Transformers to encode features and decode Gaussian primitive parameters. DepthSplat [47] leverages monocular depth to recover 3D details from sparse inputs. Other frameworks [48, 49, 50] combine Transformer or Multi-View Stereo (MVS) [51] methods to build geometric cost volumes, enabling real-time generalization. However, due to the insufficient diversity of available 3D datasets, the generalization performance of these methods remains to be further improved.

II-B Novel View Synthesis for Dynamic Scenes

Novel view synthesis for dynamic scenes naturally extends static models, with early approaches building on NeRF [52, 53, 54, 55, 56, 57, 11] and 3DGS [1], leveraging their efficient rendering capabilities for dynamic scene reconstruction. While Gaussian-based methods [2, 3, 4, 5, 6, 7, 8, 9] learn temporal attributes to model dynamic scenes as unified representations and improve reconstruction quality, their requirement to load all data simultaneously results in high memory usage, limiting their feasibility for long-sequence streaming.

To address these challenges, streaming-based methods, such as ReRF [58], NeRFPlayer [59], and StreamRF [60] reformulate dynamic scene reconstruction as an online problem. Moreover, 3DGStream [10] utilizes Gaussian-based representations combined with Neural Transformation Caches to model inter-frame motion, though it still requires over 10 seconds per frame. HiCoM [61] introduces an online reconstruction pipeline for multi-view video streams employing perturbation-based smoothing for robust initialization and hierarchical motion coherence mechanisms. Instant Gaussian Stream (IGS) [62] proposes enables single-pass motion computation guided by keyframes, reducing error accumulation and achieving reconstruction times around 4 seconds per frame. Alternative frame-tracking methods [63, 64] track Gaussian evolution across frames, supporting streaming protocols but incurring substantial per-frame data overhead.

In contrast to existing methods that require full sequence processing or incur significant per-frame optimization overhead, we propose Scale-GS, a scalable Gaussian splatting framework for efficient streaming rendering. By constructing multi-scale Gaussian representations combined with selective training, Scale-GS enables on-the-fly novel view synthesis while preserving high rendering quality.

III Method

In this section, we first introduce the preliminaries in Sec. III-A. Later we present the key pipeline of Scale-GS. The framework of Scale-GS  is presented as in Fig. 2. After decompositing the dynamic part, Scale-GS follows an anchor-based multi-scale Gaussian representation (Sec. III-B) as its core framework. We apply hybrid deformation-spawning Gaussian optimization (Sec. III-C) to model inter-frame motion. When deformation is insufficient to capture dynamics, we activate the next scale and selectively spawn new Gaussians. Once all scales have converged, redundant Gaussians are pruned to optimize the representation (Sec. III-D). In addition,  Scale-GS employs bidirectional bidirectional adaptive masking (Sec. III-E) to identify dynamic anchors and select informative viewpoints.

III-A Preliminaries

3DGS uses a dense set of Gaussian spheres to represent the whole space, and renders viewpoints via differentiable splatting combined with tile-based rasterization of these Gaussian components. For each Gaussian sphere $i$, the expectation position $\mu_{i}$ and variance $\Sigma_{i}$ determine the formation of the Gaussian $G_{i}(x)$. The point $x$ on the Gaussian sphere $G_{i}(x)$ is noted as

[TABLE]

where $\Sigma_{i}$ is composed of a rotation matrix $R_{i}\in\mathbb{R}^{3\times 3}$ and a diagonal scale matrix $S_{i}\in\mathbb{R}^{3\times 3}$, denoted as $\Sigma_{i}=R_{i}S_{i}S_{i}^{T}R_{i}^{T}$. The color and opacity of Gaussian sphere $i$ are denoted as $c_{i}$ and $\alpha_{i}$.

We use $x^{\prime}$ to define the 2D projection pixel position, and the pixel value $C\in\mathbb{R}^{3}$ is rendered via $\alpha$-composite blending. Specifically, we assume the light ray projected onto a pixel $x^{\prime}$ intersects with $N$ Gaussian surfaces along its path. The color $C(x^{\prime})$ is defined as

[TABLE]

where $N$ Gaussians are sorted from near to far and $\alpha_{i}$ signifies the opacity of Gaussian $G_{i}(x)$. With the differentiable rasterizer, all attributes of the 3D Gaussians become learnable and can be directly optimized in an end-to-end manner through training view reconstruction.

To enable structural rendering in the Gaussian splatting model, we introduce an anchor-based mechanism[28, 33]. The 3D space is uniformly partitioned into multiple voxels, with each voxel assigned a dedicated anchor responsible for managing all Gaussian primitives within its region.

The anchor-based framework is initialized by voxelizing the sparse point cloud generated from Structure-from-Motion (SfM) pipelines. Let $v$ denote the index of a specific anchor, and $V$ represent the complete set of anchors across the entire space. Each $v\in V$ corresponds to a local context feature $\hat{f}_{v}$, a 3D scaling factor $c_{v}$, and $k$ learnable offsets $O_{v}\in\mathbb{R}^{k\times 3}$ to identify the attribute of the anchor $v$ and $k$ corresponding Gaussians (indexed from [math] to $k-1$). Specifically, given the location $x_{v}$ of anchor $v$, the positions of Gaussians winthin the anchor $v$ are derived as

[TABLE]

Since the Gaussians associated with the same anchor share similar attributes, the other factors of Gaussians could be predicted by lightweight MLPs $F_{\alpha},F_{c},F_{q},F_{s}$ taking the anchor attributes as input. Specifically, we denote the relative viewing distance from anchor $v$ to the camera as $\delta_{v}$ and the viewing direction from anchor $v$ to the camera as $\vec{d}_{v}$. Taking opacity prediction as an example, the opacity values of all $k$ Gaussians within anchor $v$ are derived as

[TABLE]

The color, rotation, and scale predictions follow similar formulations using their respective MLPs $F_{c}$, $F_{q}$, and $F_{s}$.

By partitioning the whole space into voxels and assigning an anchor to each voxel, the anchor-based approach facilitates localized organization and efficient indexing of Gaussian distributions, significantly reducing the overhead of traversing and computing irrelevant Gaussians. Moreover, the neural Gaussians, which infer all Gaussian factors from the anchor attribute improves the efficiency of training.

III-B Anchor-based Multi-Scale Gaussian Representation

Variations in texture detail of the 3D scene lead to Gaussian representations of differing granularity. To improve computational efficiency, we introduce a multi-scale Gaussian optimization framework, drawing inspiration from scalable video encoding[65, 66]. In this framework, coarse-scale Gaussians are first optimized using low-resolution inputs, followed by the refinement of fine-scale Gaussians guided by high-resolution images. We separate static and dynamic regions and observe the dynamic parts. When inter-frame variations arise, optimization starts at the coarse scale and progressively refines finer scales, ensuring global structures and local details, as shown in Fig. 2(a).

We define a multi-scale structure with $L$ levels of scale and $l$ corresponds to a specific level. The $M$ training viewpoint at time $t$ with original resolution as $I_{0,t},...,I_{M-1,t}$. The viewpoint resolution at the level $l$ is denoted as $I^{l}_{0,t},...,I^{l}_{M-1,t}$. At each level $l$, the corresponding set of Gaussians $\mathcal{G}^{(l)}$ is supervised by viewpoints $I^{l}_{0,t},...,I^{l}_{M-1,t}$. Considering that variations in Gaussian representations between temporally adjacent frames of the same scene are limited, the distribution learned from the initial frame can reasonably approximate that of all frames. Thus, we initialize the 3DGS on the first frame to estimate the scale of each Gaussian. We define the maximum scale $s_{\text{max}}^{(0)}$, minimum $s_{\text{min}}^{(0)}$, and mean $s_{\text{mean}}^{(0)}$ at each level $l$.

Given that Gaussian spheres with larger scales encode less fine-grained detail, we employ a binary partitioning strategy to divide the scale space. Specifically, once the scale range for a given level is established, the subsequent level is recursively defined within the finer half of the current level’s range. Specifically, we assume the size of level $l$ is defined as $[s_{\text{min}}^{(l)},s_{\text{max}}^{(l)}]$. If the level $l+1$ is required, the scale of level $l$ is revised to $[s_{\text{mean}}^{(l)},s_{\text{max}}^{(l)}]$ and the the scale of level $l+1$ is revised to $[s_{\text{min}}^{(l)},s_{\text{mean}}^{(l)}]$, i.e, $[s_{\text{min}}^{(l+1)},s_{\text{max}}^{(l+1)}]\leftarrow[s_{\text{min}}^{(l)},s_{\text{mean}}^{(l)}],\quad[s_{\text{min}}^{(l)},s_{\text{max}}^{(l)}]\leftarrow[s_{\text{mean}}^{(l)},s_{\text{max}}^{(l)}]$. This ensures that higher-indexed levels (higher resolution) consistently receive smaller Gaussian scale ranges.

We introduce a clamp function to ensure that Gaussians at each level are constrained within their designated scale ranges, which is represented as $\text{clamp}(,)$, to enforce both upper and lower bounds on the scale of the Gaussians, i.e.,

[TABLE]

According to Eq. (5), scale parameters at each level are constrained within their designated ranges, ensuring that coarser and finer Gaussians are optimized independently.

III-C Hybrid Deformation-spawning Gaussian Optimization

As the scene changes over time, existing methods rely on either deformation, which lacks expressiveness for complex dynamics, or spawning, which is computationally inefficient due to the need for many new Gaussians. To overcome the limitations, we propose a hybrid approach that combines deformation and spawning policy. After performing deformation at scale level $l$, unresolved dynamics—either from higher-resolution variations or newly emerged components—are addressed by guided spawning at the current scale and activating deformation at the $l+1$, as shown in Fig. 2(b).

We use the anchor $v$ as an example to present the following description. For scale level $l$, the deformation process is processed through two lightweight Multi-Layer Perceptrons (MLPs) to model the dynamics of each Gaussian from $t-1$ to $t$. We adopt a decoupled design to separately model geometry and appearance. Geometric deformation leverages spatial context via hash encoding to capture 3D structure, while photometric changes—reflecting intrinsic material properties—are processed directly without spatial encoding. The framework of Gaussian deformation is presented as Fig. 2(c). At each level $l$, an geometric deformation MLP ($\mathrm{MLP_{g}}$) takes the previous frame’s center position $\mu^{(l)}_{i,t-1}$ as input, which is encoded via multi-scale hash encoding $h(\mu^{(l)}_{i,t-1})$ to capture both local and global spatial context. The $\mathrm{MLP_{g}}$ processes the hash-encoded position to predict geometric changes

[TABLE]

The output is a 7-dimensional vector, where the first 3 dimensions represent the position offset $\Delta\mu^{(l)}_{i}$ and the last 4 dimensions denote the quaternion increment $\Delta q^{(l)}_{i}$.

The appearance deformation MLP ($\mathrm{MLP_{a}}$) receives the original color $c^{(l)}_{i,t-1}$ and opacity $\alpha^{(l)}_{i,t-1}$ as inputs, and predicts the updated values $c^{(l)}_{i,t}$ and $\alpha^{(l)}_{i,t}$, i.e.,

[TABLE]

Overall, we model temporal change of all Gaussian attributes at level $l$ via residual updates from the previous frame

[TABLE]

where $\mathrm{norm}(\cdot)$ indicates quaternion normalization to ensure unit quaternion constraints for valid rotations.

The rendering of scale level $l$ follows the volume rendering approach where each level processes its own Gaussians independently

[TABLE]

The reconstruction loss at scale level $l$ is computed with respect to the corresponding resolution images, including an $\ell_{1}$ loss and a structural similarity loss to enforce perceptual fidelity.

[TABLE]

where $\hat{C}_{n,t}$ indicates the volume rendering result of viewpoint $n$ at time $t$ with Gaussian scale level $l$.

For each anchor $v$ at level $l$, we compute the average gradients of constituent Gaussians over $d$ training iterations, denoted as $\nabla g_{v}^{(l)}$, denoted as

[TABLE]

where $G_{v}^{l}$ represents the set of Gaussians belonging to anchor $v$ at scale level $l$.

We define a level-specific gradient thresholds [28], denoted as

[TABLE]

where $Vol$ indicates the volume size of each anchor and the threshold decreases exponentially with scale level $l$, guiding progressively finer control at higher resolution levels. If $\nabla g_{v}^{(l)}>\tau_{\text{add}}^{(l)}$ after deformation at $l$, suggesting that the resolution of $l$ is not sufficient to capture the underlying dynamic changes, the Scale-GS framework triggers octree Gaussian Spawning at $l$ and deformation at $l+1$. The detail of hybrid deformation-spawning strategy across multi-scale is represented as Fig. 3.

Octree Gaussian Spawning of Scale $l$. As shown in Fig. 2(d), for the Spawning of level $l$, Gaussians are spawned at predefined anchor locations, followed by an optimization phase to adapt the newly added Gaussians to the dynamic scene. After training, these Gaussians are associated with their corresponding anchors and carried forward for overfitting in the subsequent frame.

We construct an octree-based Gaussian representation to model dynamics with minimal Gaussian usage. The octree structure enables adaptive allocation of Gaussian guided by gradient information. We refer to each region partitioned by the octree within an anchor space as a subspace. At each level $l$, we compute the mean gradient $\nabla g_{\text{v}}^{(l)}$ of all Gaussians within each subspace. If $\nabla g_{\text{v}}^{(l)}>\tau_{\text{add}}^{(l)}$, a fixed number of Gaussians are randomly assigned within the subspace, which is then recursively subdivided. This process continues until all subspaces fall below the threshold or the spatial resolution reaches one-thousandth of the original domain.

Gaussian Deformation of Scale $l+1$. To enable finer-grained deformation modeling, the optimization process activates Gaussians at the next resolution level $l{+}1$. These Gaussians are inherited from frame $t{-}1$ and share the same anchor structure as level $l$, but operate at a higher spatial resolution through hierarchical refinement.

III-D Redundant Gaussian Removing

As the hybrid deformation-spawning strategy introduces additional Gaussians in each frame, a redundant Gaussian removing step is applied after all scales converging to prevent uncontrolled growth in the number of Gaussian over time. For each Gaussian $i$ managed by anchor $v$, we define a 1-D optimizable mask $M_{v,i}$. The $M_{v,i}$ is passed through a sigmoid function $\sigma(\cdot)$ to ensure differentiability. When $M_{v,i}$ is large, $\sigma(M_{v,i})\rightarrow 1$; conversely, when $M_{v,i}$ is small, $\sigma(M_{v,i})\rightarrow 0$, effectively removing the corresponding Gaussian.

For rendering pixels covered by anchor $v$, the projection color after masking is defined as:

[TABLE]

When the sigmoid value $\sigma(M_{v,i})$ approaches zero, the corresponding Gaussian $i$ under anchor $v$ is excluded from the volume rendering process, regardless of the scale level from which it originated. Thus, the loss function is then employed to optimize the Gaussians, filtering out redundant ones. The total loss combines the reconstruction error from the multi-scale fused rendering with a sparsity regularization term, i.e.,

[TABLE]

where $\mathcal{L}^{(l)}$ represents the rendering loss on each scale, and $\lambda_{\text{r}}$ controls the redundancy removing regularization. The redundancy removing term reduces redundant Gaussians by penalizing non-zero mask values, thereby promoting a compact representation across all anchors and scale levels.

III-E Bidirectional Adaptive Masking

We propose a bidirectional adaptive masking mechanism that selects dynamic spatial anchors and informative camera views for each training frame, serving as a preprocessing step to reduce redundant computation in static regions and mitigate supervision from views with limited marginal utility.

The forward masking process selects dynamic anchors by distinguishing motion across adjacent multi-view frames. Specifically, temporal variations between two consecutive frames enable a coarse estimation of spatial motion for individual image pixels. The motion is then back-projected into 3D space to identify spatial position with significant displacement. If an anchor’s coverage consistently exhibits prominent motion patterns over multiple frames, it is designated as a dynamic anchor, which subsequently guides the optimization of dynamic objects.

The backward masking is designed to select a subset of camera views aligned with these dynamic regions. We introduce a relevance score for each camera view $c_{k}$. Let $\mathrm{IoU}(c_{k},v)$ denote the normalized intersection-over-union between the image-space projection of anchor $v$ and the field-of-view of camera $c_{k}$. Given a threshold $\tau_{\text{view}}$, we define the view relevance score as

[TABLE]

where $\mathbf{1}()$ is the indicator function and $\omega(c_{k},v)$ is a direction weight, defined as

[TABLE]

where $\mathbf{n}_{v}$ is the average normal vector of Gaussians in anchor $v$, $d_{v}$ denotes the normal direction of the viewpoint $d_{c_{k}}$ .

Given the ranking of $S(c_{k})$, the top-ranked views are selected for training, as they are considered the most relevant to the dynamics. This bidirectional masking mechanism reduces computational redundancy caused by uninformative viewpoints of dynamic scenes.

III-F Algorithmic Summarization

We summarize  Scale-GS dynamic scene optimization approach in Algorithm 1, which performs multi-scale guided dynamic updates through hybrid deformation-spawning optimization (III-C) combined with bidirectional adaptive masking (III-E).

The algorithm operates on the pre-initialized multi-resolution Gaussian hierarchy, applying the hybrid deformation-spawning optimization strategy across multiple resolution levels while using bidirectional adaptive masking to focus computation on dynamic regions and informative views. This approach ensures efficient and targeted optimization for dynamic scene reconstruction while maintaining scale-aware structure throughout the process.

Implementation Details. All experiments are conducted in a virtual environment using Python 3.9 and PyTorch 2.1.0 as the primary deep learning framework. Additional libraries include plyfile 0.8.1 for data handling, torchaudio 0.12.1 for audio processing, and torchvision 0.13.1 for computer vision tasks. Training and inference were performed on a workstation equipped with an NVIDIA RTX 4090 GPU with CUDA 12.6 support, complemented by cudatoolkit 11.8 for GPU acceleration. The proposed method is implemented in PyTorch and trained under the above configuration, ensuring seamless integration of hardware capabilities and software functionalities for reliable experimental outcomes.

IV Experiment Results

In this section, we thoroughly analyze the experiment results of Scale-GS. Specifically, we first introduce the dataset and the experimental setup. Later, we present the qualitative results and quantitative results. Finally, we conduct the ablation study to illustrate the effectiveness of each key module.

IV-A Datasets

We evaluate  Scale-GS method on three real-world dynamic scene datasets: the MeetRoom dataset[60], the NV3D (Neural 3D Video) dataset[67], and the Google Immersive Light Field Video dataset[68]. All of the datasets exhibit complex motion and occlusion patterns, thus suitable for evaluating the free-viewpoint rendering.

NV3D Dataset[67]. The Neural 3D Video dataset contains six dynamic scenes captured using a synchronized 21-camera array arranged in a semi-circular configuration. Each camera records the frames at a resolution of $2704\times 2028$ with 300 frames, including various human motion interactions under indoor lighting conditions.

MeetRoom Dataset[60]. The MeetRoom dataset provides 3 indoor dynamic scenes recorded with 13 synchronized cameras, each capturing at $1280\times 720$ resolution with 300 frames. The subjects are performing structured activities such as sitting, walking, or conversing in an office-like setting.

Google Immersive Dataset[68]. This dataset contains 15 complex dynamic scenes captured using a high-fidelity immersive light field video rig consisting of 46 time-synchronized cameras on a 92 cm diameter hemisphere, each capturing at $2560\times 1920$ resolution. The system supports a large viewing baseline (up to 80 cm) and a wide field of view ($>$220°), posing very challenging novel view synthesis scenes.

IV-B Experimental Setup

Hyper-parameter settings. In the qualitative experiment and quantitative experiment, we adopt a three-level hierarchical structure ($L=3$), corresponding to an image resolution pyramid obtained by downsampling the original input to $\{1/4,1/2,1\}$ for levels $l=1,2,3$, respectively. We set $\lambda_{\text{SSIM}}=0.2$, $\lambda_{\text{r}}=0.001$ and the level-specific gradient thresholds $\tau_{\text{add}}^{(l)}=0.01/4^{l-1}$, which yields $\{\tau_{\text{add}}^{(l)}\}=\{0.01,0.0025,0.000625\}$ for levels $l=1,2,3$ respectively.

Metrics. We assess the rendering fidelity of  Scale-GS method using three widely adopted perceptual and photometric metrics: PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and LPIPS (Learned Perceptual Image Patch Similarity). These metrics respectively reflect pixel-level accuracy, structural coherence, and perceptual similarity. Furthermore, we compare the training time and rendering time to specify the efficiency of the proposed Scale-GS.

Baselines. We evaluate  Scale-GS against five state-of-the-art dynamic scene rendering methods that represent different approaches to temporal modeling and real-time performance.

Deformable 3D Gaussians [6] extends the static 3D Gaussian splatting framework to handle dynamic scenes through a canonical space representation. The method learns a set of 3D Gaussians in canonical space and captures temporal variations using a deformation field implemented as an MLP that predicts position, rotation, and scaling offsets. To address potential jitter from inaccurate camera poses, the authors introduce an annealing smooth training mechanism, enabling both high-fidelity rendering quality and real-time performance.

4DGS [8] takes a different approach by incorporating a Gaussian deformation field network that operates on canonical 3D Gaussians. The method employs a spatial-temporal structure encoder coupled with a multi-head deformation decoder to predict Gaussian transformations across time. By modeling both Gaussian motion and shape changes through decomposed neural voxel encoding, 4DGS achieves efficient real-time rendering while maintaining temporal consistency.

3DGStream [10] focuses on streaming applications, enabling on-the-fly training for photo-realistic free-viewpoint videos. The method introduces a Neural Transformation Cache (NTC) to model 3D Gaussian transformations and employs an adaptive Gaussian spawn strategy for handling newly appearing objects. The framework operates through a two-stage pipeline: first training the NTC for existing Gaussians, then spawning and optimizing additional frame-specific Gaussians to accommodate emerging scene content.

HiCoM [61] presents a comprehensive framework specifically designed for streamable dynamic scene reconstruction. It combines three key components: a perturbation smoothing strategy for robust initial 3D Gaussian representation, a hierarchical coherent motion mechanism that captures multi-granular motions through parameter sharing within regional hierarchies, and a continual refinement process that evolves scene content while maintaining representation compactness.

IGS [62] offers a generalized streaming framework centered around an Anchor-driven Gaussian Motion Network (AGM-Net). This network projects multi-view 2D motion features into 3D space using strategically placed anchor points to drive Gaussian motion. The method further incorporates a key-frame-guided streaming strategy that refines key frames and effectively mitigates error accumulation during long sequences.

IV-C Qualitative Results

We choose 2 scenes from NV3D and the Google Immersive dataset to conduct a comprehensive qualitative evaluation, including indoor scene and outdoor scene.

IV-C1 Comparison in Indoor Scene

Fig. 4 presents the visual result on the NV3D dataset on 3 progressive frame index. In this scenario, it is critical to analyze the motion of the human and the dog and accurately reconstruct the dynamics of the flame. According to Fig. 4, Scale-GS reconstructs significantly sharper and accurate fine-grained details compared to the baselines.

Specifically, in the frame 2, the 4DGS and 3DGStream fail to capture the trailing motion of hands, exhibiting noticeable blur. Meanwhile, four baselines render blurred occluded backgrounds, showing a large difference from the ground truth. The proposed Scale-GS not only reconstructs the clear texture of the blender, but renders the contour of the occluded object. In the frame 151, 4DGS and IGS fail to properly render the dog’s eyes. The 3DGStream and HiCom display obvious abnormal Gaussian points in the dog’s head area. Compared to the baselines, the Scale-GS renders a richly furred dog head without introducing floaters. The frame 183 indicates that the Scale-GS accurately captures the natural shape variations of the flame and intricate details. Although the baselines achieve flame rendering from a global perspective, they do not match the ground truth in the zoomed-in regions.

IV-C2 Comparison in Outdoor Scene

Fig. 5 presents outdoor comparative results on the Google Immersive dataset. Compared to indoor scenes, outdoor scenes cover a larger area, involve more rendering details, and are more challenging. The red bounding boxes highlight regions showing the dog’s head. In addition to reconstructing the changes in the dog’s facial features and head, it is necessary to render the complex fur.

According to Fig. 5, the proposed Scale-GS reconstruct clear and consistent face of the dog, while the baselines fail to accurately represent the dog’s facial contours. Specifically, 4DGS, 3DGStream, and HiCom exhibit strong blurring around the dog’s nose area in the frame 18. Moreover, the 3DGStream and HiCom methods display abnormal Gaussian representations at the dog’s eyebrows. In contrast, the subtle details around the eyes and nose of the Scale-GS are preserved with high fidelity, especially maintaining consistent quality across different viewing angles.

Overall, Fig. 4 and Fig. 5 demonstrate that Scale-GS achieves superior visual quality and faithfulness compared to the baselines, particularly in challenging scenarios involving complex human interactions, dynamic elements like flame, and intricate textures such as animal fur.

IV-D Quantitative Results

Tab. I presents the quantitative comparison to baselines on the 3 different datasets. We evaluate the novel view quality and rendering efficiency. To ensure a fair comparison, all competing methods are evaluated using the same set of Gaussians initialized from the [math]-th frame, and the same variant of Gaussian splatting rasterization is applied consistently across all approaches.

According to Table I, Scale-GS outperforms the baselines and achieves the best reconstruction quality with the fastest training speed. In the rendering quality aspect, Scale-GS achieves a PSNR improvement from 33.62dB (second-best IGS) to 34.47dB on NV3D dataset, from 30.13dB to 31.58dB on Meeting Room dataset, and from 29.72dB to 31.18dB on Google Immersive dataset, demonstrating consistent quality enhancement across all evaluation scenarios. For SSIM metrics, Scale-GS achieves competitive performance with 0.912 on Google Immersive dataset (best), 0.966 on NV3D dataset (0.001 below the best), and 0.908 on Meeting Room dataset (0.001 below the best), indicating excellent structural preservation. The LPIPS scores show perceptual quality improvements, with  Scale-GS achieving the best result of 0.142 on Meeting Room dataset and maintaining competitive performance on other datasets.

The sub-optimal performance of competing methods can be attributed to several limitations: (1) Global deformation inefficiency: Methods like Deform and 4DGS apply deformation to all Gaussians uniformly, leading to unnecessary computations on static regions and reduced optimization focus on truly dynamic areas. (2) Single-scale representation constraints: Traditional approaches like 3DG-S and HiCoM operate at fixed resolutions, failing to leverage hierarchical optimization strategies. In contrast, our multi-scale approach enables rapid convergence by first focusing on coarse-scale Gaussians that capture major scene changes, then progressively refining finer-scale Gaussians with increasing precision. This hierarchical refinement allows the optimization to quickly identify and target only the Gaussians that require updates at each scale level, resulting in accelerated training convergence and progressively improved rendering quality as finer details are incorporated. (3) Lack of adaptive supervision: Existing methods fail to adaptively select the most informative views and spatial regions during training, resulting in suboptimal resource allocation and slower convergence. In contrast, our multi-scale framework with hybrid deformation-spawning policy and bidirectional adaptive masking addresses these limitations systematically, enabling both superior reconstruction quality and computational efficiency.

IV-E Ablation Study

IV-E1 The Evaluation of Scale Number

We conduct a group of ablation study to investigate the influence of the scale number of Scale-GS. We choose the Face Print scene from the Google Immersive Dataset and set the scale to 2, 3, 4, and 5, respectively. The result of average SSIM, PSNR, LPIPS, and training time is demonstrated in Table II.

The experimental results indicate that the configuration with 3 scales achieves the best performance across all metrics, yielding an SSIM of 0.916, a PSNR of 31.233 dB, and an LPIPS score of 0.154. It is noted that with the increase of the scale number, the rendering quality may decline under the same training iterations. This is because the redundant scale requires training computation, thus the convergence of each scale becomes insufficient. Therefore, 3 level of scales strike the tradeoff between visual quality and training efficiency.

To further validate the visual performance gap across different scale settings, in Fig. 6, we present a qualitative comparison of the zoom-in region in the same scene. It can be observed that the 3-scale configuration produces the most faithful reconstruction, particularly in facial regions such as the eyes, eyelashes, and hair strands, closely resembling the ground truth. The hand region near the drawing board is reconstructed the clearest details with the scale number as 3. In comparison, other scale configurations exhibit varying degrees of Gaussian transparency or blurring in the hand area, which negatively affects the rendering quality.

IV-E2 The Effectiveness of Hybrid Deformation-spawning

To evaluate the effectiveness of the hybrid deformation-spawning strategy, we conduct an ablation study on the MeetRoom dataset by comparing three variants: (1) the full hybrid strategy, (2) w/o deformation, and (3) w/o spawn. The results, summarized in Table III, show that the hybrid approach achieves the best performance across all evaluated metrics, in which SSIM is 0.913, PSNR is 31.602 dB, and LPIPS is 0.138, and the least training time per frame (2.9 seconds). In contrast, removing either component leads to a noticeable degradation in both reconstruction quality and efficiency.

Furthermore, we demonstrate the visual quality(top) and the statistical PSNR and training time(bottom) comparison among the full hybrid strategy, without deformation, and without spawning in Fig. 7. According to the visual result, the full hybrid strategy exhibits clear edge and accurate color representation. Moreover, the details of the plant leaves and the fabric textures are also well preserved. These visual results further validate that the hybrid mechanism achieves high-quality rendering and efficient training of dynamic scenes.

As shown in Fig. 7, our hybrid approach consistently achieves the shortest training time per frame (averaging around 3.2 seconds), while both ablated variants require significantly longer training periods. Specifically, the “w/o spawn” variant, though taking more time than our method, has relatively more stable training duration compared to “w/o deformation”; the “w/o deformation” variant shows considerable training time fluctuations throughout the sequence and exhibits the highest training costs. Fig. 7(b) demonstrates the PSNR evolution across frames, where our method maintains relatively stable and high PSNR values (around 34.5 dB). The “w/o spawn” variant can keep a decent level of PSNR for a while but shows more pronounced fluctuations than our method as frames progress, while the “w/o deformation” variant experiences a more gradual yet evident decline in PSNR over time and overall lower reconstruction quality. The results demonstrate that the Scale-GS method not only trains faster but also maintains more stable rendering quality throughout the sequence.

Even the dynamic nature of the scene leads to unstable PSNR variations, the hybrid deformation and spawning strategy performs the best in both visual quality and training efficiency in most cases. This suggests that the joint optimization of deformation and spawning enables more efficient and stable learning of dynamic scene representations.

IV-E3 Viewpoint Selection

To validate the effectiveness of Bidirectional Adaptive Masking (BAM) mechanism, we conduct ablation studies on five challenging scenes from the Google Immersive Dataset: Car, Goats, Dogs, Face Paint, and Welder. These scenes feature diverse dynamic content ranging from fast-moving objects to complex deformations, providing a comprehensive testbed for evaluating view selection strategy.

The quantitative results are presented in Table IV.  Scale-GS BAM-based view selection consistently outperforms the baseline across all tested scenes, achieving improvements of 0.7-1.7 dB in PSNR and 0.007-0.035 in SSIM. The most significant improvement is observed in the Welder scene, where BAM selection achieves 30.281 dB PSNR compared to 28.552 dB without selection, representing a substantial 1.729 dB gain. This scene benefits particularly due to its complex welding sparks and rapid lighting changes, where targeted view selection effectively focuses learning on the most informative perspectives.

The improvement across diverse scene types demonstrates that BAM successfully identifies and prioritizes views that provide meaningful supervision for dynamic regions. By filtering out redundant or less informative viewpoints, the BAM not only improves reconstruction quality but also enhances training efficiency. The bidirectional masking—selecting both spatial anchors and camera views—proves essential for handling the complexity of multi-view dynamic scene reconstruction.

These results validate that bidirectional view selection mechanism effectively ensure that Gaussians receive supervision from the most relevant camera perspectives, leading to more accurate and stable dynamic scene modeling.

V Conclusion and Future Work.

Conclusion. In this work, we present Scale-GS, a scalable Gaussian Splatting framework for efficient and redundancy-aware dynamic scene training. The proposed Scale-GS proposes the anchor-based multi-scale Gaussian representation integrating a hybrid deformation–spawning optimization, redundant Gaussian removing, and a bidirectional adaptive masking module. Our method effectively reduces the computational overhead of existing approaches by filtering static or irrelevant Gaussian spheres. The hybrid deformation–spawning strategy preserves the structured inference from deformation while enhancing the model’s capacity to represent large-scale dynamic motions. Extensive experiments demonstrate that Scale-GS achieves superior visual fidelity and significantly accelerates training. The proposed framework opens up promising opportunities for remote immersive experiences such as VR, AR, and immersive video conferencing.

Future work. To further reduce the redundant computation and storage, we aim to incorporate semantic understanding into the multi-scale representation to improve the representation efficiency of Gaussian spheres. The semantic information guides the allocation and prioritization of Gaussian spheres by adapting their scale, density, and training frequency according to the semantic importance and structural complexity of different areas. Furthermore, integrating Scale-GS with neural compression could further enhance scalability and enable deployment in bandwidth and resource-constrained environments.