Global Motion Corresponder for 3D Point-Based Scene Interpolation under Large Motion

TL;DR

This paper introduces GMC, a novel method for 3D scene interpolation that effectively handles large motions by learning SE(3) mappings, resulting in smoother transitions and better extrapolation compared to existing methods.

Contribution

GMC is the first approach to robustly interpolate 3D scenes with large global motions by learning unary potential fields into a shared canonical space.

Findings

Outperforms existing methods on large-motion scene interpolation

Enables extrapolation beyond observed states

Achieves smoother and more accurate scene transitions

Abstract

Existing dynamic scene interpolation methods typically assume that the motion between consecutive timesteps is small enough so that displacements can be locally approximated by linear models. In practice, even slight deviations from this small-motion assumption can cause conventional techniques to fail. In this paper, we introduce Global Motion Corresponder (GMC), a novel approach that robustly handles large motion and achieves smooth transitions. GMC learns unary potential fields that predict SE(3) mappings into a shared canonical space, balancing correspondence, spatial and semantic smoothness, and local rigidity. We demonstrate that our method significantly outperforms existing baselines on 3D scene interpolation when the two states undergo large global motions. Furthermore, our method enables extrapolation capabilities where other baseline methods cannot.

Equations

$$D_{i,j}^{\text{ toy}}$$

$$+w_{\mu }\Vert {\mathbf{\mu }}_i-{\mathbf{\mu }}_j{\Vert }_2^2.$$

$$\underset{{\hat{\mathbf{\mu }}}_i^{(0)}}{\underbrace{{\mathbf{R}}_i^{(0)}{\mathbf{\mu }}_i^{(0)}+{\mathbf{t}}_i^{(0)}}}=\underset{{\hat{\mathbf{\mu }}}_j^{(1)}}{\underbrace{{\mathbf{R}}_j^{(1)}{\mathbf{\mu }}_j^{(1)}+{\mathbf{t}}_j^{(1)}}},$$

$$\mathcal{F}(\tilde{\mathbf{f}},\mathbf{\mu })=(\mathbf{R},\mathbf{t}).$$

$$E_{i,j}$$

$$+w_{\mu }\Vert {\hat{\mathbf{\mu }}}_i-{\hat{\mathbf{\mu }}}_j{\Vert }_2^2,$$

$${\mathcal{L}}_{\text{E}}=\sum _{g_i\in {\mathcal{G}}_0}\underset{g_j\in {\mathcal{G}}_1}{\min }{E}_{i,j}+\sum _{g_j\in {\mathcal{G}}_1}\underset{g_i\in {\mathcal{G}}_0}{\min }{E}_{j,i}.$$

$${\mathbf{\mu }}_i^{(0),t=1}$$

$$={\left({\mathbf{R}}_j^{(1)}\right)}^{\top }\left({\mathbf{R}}_i^{(0)}{\mathbf{\mu }}_i^{(0)}+{\mathbf{t}}_i^{(0)}-{\mathbf{t}}_j^{(1)}\right).$$

$$\frac{1}{kN}\sum _{g_i\in {\mathcal{G}}_k}\sum _{g_j\in {\mathrm{NN}}_i}\left|\Vert {\mathbf{\mu }}_i-{\mathbf{\mu }}_j{\Vert }_2^2-\Vert {\hat{\mathbf{\mu }}}_i-{\hat{\mathbf{\mu }}}_j{\Vert }_2^2\right|,$$

$$\mathcal{L}={\mathcal{L}}_{\text{E}}+\alpha {\mathcal{L}}_{\text{iso}},$$

$${\mathcal{L}}_{\mathrm{render}}=\beta {\mathcal{L}}_{\mathrm{RGB}}({\mathbf{I}}_0,{\hat{\mathbf{I}}}_0)+{\mathcal{L}}_{\mathrm{RGB}}({\mathbf{I}}_1,{\hat{\mathbf{I}}}_1),$$

Full text

[TABLE]

1 Introduction

Dynamic scene interpolation reconstructs continuous motion from two sets of discrete multi-view frames, which is a fundamental challenge in computer vision driven by the increasing demand for photorealistic animation. Recent methods built on point-based representations [12, 34] are especially appealing as they enable efficient training and real-time rendering.

The core challenge in scene interpolation with point-based representations lies in establishing reliable correspondences: each point must predict its motion to a corresponding location in another frame. Most existing methods rely on a critical assumption that the motion between adjacent timesteps is small enough that point positions do not change significantly. Under this assumption, determining correspondence reduces to matching points within small local neighborhoods, which has been successfully explored by existing works [25, 28, 17, 20]. However, this small-motion assumption does not always hold in practice. Many real-world scenarios, such as recording athletes in motion, vehicles on a highway, or any dynamic scene where temporally dense capture is expensive or infeasible, routinely violate this fundamental assumption.

When motion becomes sufficiently large, dynamic scene interpolation faces a fundamental breakdown. As shown in Figure 2, existing works that rely on the motion locality assumption would fail. This failure stems from the ill-posed nature of point correspondence under a large motion: 1) a point’s local neighborhood becomes unreliable as objects move far from their original positions, and 2) globally, a point may have multiple plausible matches, making the point-to-point correspondence ambiguous. As illustrated in Figure 1, naïvely matching nearest neighbors in these cases results in criss-cross matches, making it unusable for scene interpolation.

We introduce GMC to address this fundamental limitation by learning smooth global correspondences through transformation to a shared canonical space for each timestep. Our key insight is to replace direct point-to-point matching with unary potential fields that predict SE(3) transformations for each timestep. These fields take universal features such as DINO [4] as input and leverage the inductive bias of MLPs to address the ill-posed nature of global matching. By construction, this approach ensures that semantically similar points move coherently by predicting coherent transformation.

Our approach seamlessly integrates point-based representations, such as 3D Gaussian Splatting [12] (3DGS) with continuous and queryable SE(3) fields, enabling robust interpolation and extrapolation under large motion. Specifically, we train two unary potential fields alongside two sets of 3D Gaussians, where each field learns SE(3) transformations that map its respective Gaussians into a shared canonical space. By defining the GMC over both Gaussian 3D coordinates and semantic features, our SE(3) fields exhibit smoothness and continuity across both space and semantics. Forward and backward mapping through the canonical space yields dense per-point trajectories for both interpolation and extrapolation.

We evaluate GMC on interpolation using eight scenes from the PAPR In Motion dataset [20] and twelve additional challenging scenes with multi-object interactions and large motions. Results demonstrate that GMC significantly outperforms baseline methods under large motion conditions. Additionally, GMC enables realistic motion extrapolation, a capability lacking in existing baselines. We also provide preliminary results showing that our method can not only better reason in the setting of a sparse temporal capture but can also improve reconstruction in sparse spatial capture.

In summary, our contributions are:

• •

Global Motion Corresponder (GMC), a novel method for 3D scene interpolation under large motion.

• •

Comprehensive experimental results demonstrating GMC’s superior performance over prior methods for interpolation on challenging large-motion scenarios.

• •

Further demonstration of GMC’s capabilities in enabling extrapolation and improving reconstruction quality under sparse temporal and spatial capture conditions.

2 Related Work

3DGS [12] is a point-based representation that rasterizes small differentiable Gaussians for novel views synthesis of a 3D scene. Unlike volumetric representations like Neural Radiance Fields [18], which require querying multiple points along each ray, 3DGS represents pixels using only a few Gaussians, achieving faster training and real-time rendering.

The challenge of dynamic scene interpolation becomes particularly acute when scenes involve complex and large motions across multiple objects. Traditional point cloud interpolation methods [35, 30, 33, 16] fail under such conditions. Numerous 3DGS-based approaches [15, 6, 9, 3, 17, 37, 14, 26, 25, 28] have emerged to tackle dynamic scene interpolation under the small-motion assumption. However, our problem requires both high-quality scene reconstruction and smooth interpolation, demanding robust performance when this assumption breaks down. Current 3DGS-based approaches fall into two categories: deformation fields and iterative refinement.

Scene Interpolation with Deformation Fields.

These methods use neural deformation fields to predict pre-Gaussian displacement at given timesteps, jointly optimizing deformation parameters alongside 3D Gaussian models. Representative works include 4DGS [25] and Deformable 3DGS [28]. However, under large motion, correspondence estimation across timesteps becomes fundamentally ill-posed. Hence, the training of the deformation network is very unstable, resulting in poor reconstruction quality and implausible motion estimates.

Scene Interpolation by Iterative Refinement.

These approaches achieve interpolation by iteratively refining point primitives from previous timesteps. Under small-motion assumptions, point geometry and semantics do not change drastically between timesteps. These methods pre-train 3D scene representations at initial states, then update the point positions and semantics via rendering loss gradients at subsequent timesteps, storing model weights at each timestep. Interpolation proceeds by blending adjacent timestep weights. Notable examples include Dynamic Gaussian [17] and PAPR In Motion [20]. However, under large motions, simple position fine-tuning from previous states proves insufficient, causing these methods to fail.

Visual Features for Correspondence.

Features extracted from self-supervised vision transformers [8, 4] serve as powerful visual descriptors that preserve local semantic information. These features excel in downstream tasks such as zero-shot segmentation and semantic correspondence [2]. Zhang et al.  [31] demonstrate semantic correspondence using features from large vision models such as DINO [4] and Stable Diffusion [21, 5]. However, while achieving impressive 2D correspondence results, these methods fail to extend to 3D representations or dynamic scenes due to their lack of 3D awareness under complex object motions and occlusion. To address this issue, Zero-Shot 3D Shape Correspondence [1] proposes a method for establishing correspondence between 3D shapes without explicit supervision. Although innovative, the method is not specifically designed to capture temporal dynamics. In contrast, our method leverages DINO [4] features to learn motion-aware point-based correspondences, uniquely enabling both 3D scene interpolation and extrapolation under large motion.

3 Method

In the previous sections, we explained that traditional methods assume spatial positions of 3D points change minimally between adjacent timesteps, which holds only under small, local motion. Since each Gaussian’s position remains nearly constant, it is easy to establish correspondences between points from adjacent timesteps by simply identifying the Euclidean nearest neighbors. Using these correspondences, existing methods either refine point positions from previous timesteps or train neural deformation fields to predict per-point displacements.

However, nearest-neighbor correspondence fails when scenes undergo substantial motion. Under this premise, finding correspondences between adjacent timesteps is not as straightforward. When correspondences rely solely on spatial proximity, severe mismatches can occur (Figure 1), causing existing methods to produce implausible trajectories that violate spatial rigidity and physical constraints (Figure 2). Hence, the core challenge under large motion is actually analogous to the challenge of establishing smooth global correspondences between primitives across time.

Although establishing global correspondences remains challenging for current works in literature, it is intuitive for human perception. Humans can effortlessly perceive the geometry and physical properties of dynamic 3D scenes, intuitively predicting intermediate or future motions from just two timesteps. This is because humans have inherent semantic knowledge about the physical world and understanding of local spatial rigidity. The latter property is important, which means that even though the overall motion may be non-rigid, the movement of nearby object parts move similarly.

3.1 Challenges in Learning Correspondence

Since a naive nearest-neighbor search in Euclidean space using point coordinates does not work under the large motion setup, we need to find a way to conduct a global search to obtain point-to-point correspondences. For instance, we can try to define a function that allows us to measure the distance between points of two arbitrary Gaussian $g_{i}$ and $g_{j}$. One option is to use a weighted sum of color ($\boldsymbol{c}$), semantic ($\boldsymbol{f}$), and spatial ($\boldsymbol{\mu}$) features to define the following distance function:

[TABLE]

It can be reasoned that even this distance function is not enough to establish reliable correspondences. This is because there may be many Gaussian elements within a scene that can have the same color and semantic features. For instance, consider point “a” in column 2 of Figure 1: by the distance function defined in Eq. 1, its closest match in timestep 1 is point “b”, because both of them have the same color and semantic features. In general, it is difficult to find analytical distance metrics that yield the correct correspondences.

Since it is hard to come up with a pre-defined analytical function, we propose to learn these correspondences through a differentiable optimization-based method. Before describing our approach, we first highlight key properties essential for tackling the challenges posed in establishing correspondences and consequently recovering the scene motion under the large-motion assumption:

The semantics and color of corresponding points should be in agreement. 2. 2.

Semantically similar points in a neighborhood should move coherently.

3.2 Global Motion Corresponder

We give a quick overview of our approach to tackle the challenge of finding smooth and accurate global correspondence (Figure 4 and 5). We propose transforming both sets of Gaussians into a learnable shared canonical space where corresponding Gaussians occupy identical spatial locations:

[TABLE]

where $\boldsymbol{R}$ and $\boldsymbol{t}$ represent learnable point-wise transformations for each timestep. The transformations $\left(\boldsymbol{R}^{(0)}_{i},\boldsymbol{t}^{(0)}_{i}\right)$ and $\left(\boldsymbol{R}^{(1)}_{j},\boldsymbol{t}^{(1)}_{j}\right)$ are obtained from our Unary Potential Fields $\mathcal{F}_{0}$ and $\mathcal{F}_{1}$, which are parameterized as MLPs. The parameters of these MLPs are optimized using our proposed Energy-based loss. Now, we go into the technical details of the design of our Unary Potential Fields and Energy-based loss.

3.2.1 Unary Potential Field

We define a single Unary Potential Field $\mathcal{F}$, as function that maps 3D coordinates $\boldsymbol{\mu}\in\mathbb{R}^{3}$ and feature vectors $\tilde{\boldsymbol{f}}\in\mathbb{R}^{4}$ to SE(3) transformations. It outputs a quaternion representation $\boldsymbol{R}\in\mathbb{R}^{4}$ and translation $\boldsymbol{t}\in\mathbb{R}^{3}$:

[TABLE]

In practice, we use PCA-projected DINO features for $\tilde{\boldsymbol{f}}$ to capture semantic information efficiently, and $\boldsymbol{R}$ is converted to a $3\times 3$ rotation matrix when applied in subsequent transformations.

It is a well-known fact that neural networks are able to learn smooth functions such that the output does not change abruptly. Since we wish to keep the predicted motion smooth with respect to both semantic features and spatial positions, we leverage the inductive bias of MLPs for our unary potential. Hence, $\mathcal{F}$ is parameterized as an MLP. Since $\mathcal{F}$ takes features $\tilde{\boldsymbol{f}}$ as input, semantically similar points will naturally predict similar transformations by the inductive bias of the MLP.

We term these potential fields “unary” because each timestep requires individual mapping to the canonical space. The same object at different timesteps needs distinct transformations to the shared canonical space. Hence, we need to have separate MLPs per timestep ($\mathcal{F}_{0},\mathcal{F}_{1}$).

3.2.2 Energy-based Loss

We now define our energy-based loss used to optimize unary potential fields for global correspondence learning. Concretely, using the means of Gaussians mapped from $\mathcal{G}_{0}$ and $\mathcal{G}_{1}$ to the canonical space, we have:

[TABLE]

where $\hat{\boldsymbol{\mu}}_{i}$ and $\hat{\boldsymbol{\mu}}_{j}$ represent the transformed means in the canonical space. Comparing this term with Eq. 1, rather than using spatial distance in the original Euclidean space, we are now using spatial distance in the canonical space.

To train the parameters of $\mathcal{F}_{0}$ and $\mathcal{F}_{1}$, we define a bidirectional loss using the aforementioned energy term:

[TABLE]

This loss encourages Gaussians that are similar in color and feature space, and that are close in the canonical space, to be matched, prompting smooth and realistic correspondences. It is important to note two crucial designs:

The bidirectional loss that operates on both sets of Gaussians $\mathcal{G}_{0},\mathcal{G}_{1}$ is essential. Optimizing the energy term for just one timestep can lead to a scenario similar to column 2 of Figure 1, where the back of the pencil at timestep 1 does not have any correspondence. The two-way nature of the loss ensures that all points in both timesteps find correspondences. 2. 2.

Color $\boldsymbol{c}$ and features $\boldsymbol{f}$ terms in Eq. 4 are crucial. Since unary potential fields are randomly initialized, the contribution of $w_{\mu}\|\hat{\boldsymbol{\mu}}_{i}-\hat{\boldsymbol{\mu}}_{j}\|_{2}^{2}$ at the beginning of the training is arbitrary. Hence, the color and feature terms help to establish initial “soft” correspondences.

3.2.3 Global Motion

Given learned unary potentials, we can then recover the global motion for Gaussian $g^{0}_{i}$ in $\mathcal{G}_{0}$ with position $\boldsymbol{\mu}^{0}_{i}$ by first mapping it to the shared canonical space using its learned unary potential field $\mathcal{F}_{0}$: $\hat{\boldsymbol{\mu}}^{(0)}_{i}=\boldsymbol{R}^{(0)}_{i}\boldsymbol{\mu}^{(0)}_{i}+\boldsymbol{t}^{(0)}_{i}$. Next, we can find its corresponding Gaussian $g_{j}$ in $\mathcal{G}_{1}$ using the energy term defined in Eq 4 in the shared canonical space. We refer to the mean of $g_{j}$ as $\boldsymbol{\mu}^{(1)}_{j}$. Finally, we can see the inverse transformation from its learned unary potential field $\mathcal{F}_{1}$ (refer Eq. 2):

[TABLE]

The overall global motion for $g^{0}_{i}$ is then the displacement vector $|\boldsymbol{\mu}_{i}^{(0),t=1}-\boldsymbol{\mu}_{i}^{(0)}|$.

3.3 Training Details

Local Isometry Loss.

To further enforce the smoothness and rigidity, we incorporate a local isometry loss applied to the transformations into the canonical frame, inspired by prior works [23, 17, 20]. For each Gaussian $g_{i}$, we find its $k$ nearest neighbors $\mathrm{NN}_{i}$ in the original space and enforce that distances are preserved after transformation with Local Isometry Loss $\mathcal{L}_{\text{iso}}=$

[TABLE]

where $N$ is the total number of Gaussians. This loss penalizes changes in local geometric relationships, promoting locally rigid transformations. In practice, we apply this loss to the transformation into the canonical space, as well as the full transformation between frames $t=0$ and $t=1$ (see Sec. 3.4 for this full transformation). The total loss for training each GMC combines the energy and the local isometry terms:

[TABLE]

where $\alpha$ is a weight that balances the two terms. We gradually increase $\alpha$ during training, starting from zero, to allow the model to first establish correspondences before enforcing rigidity.

Joint Refinement.

After GMC learning, we perform a joint refinement to update both GMC and the Gaussian sets $\mathcal{G}_{0}$ and $\mathcal{G}_{1}$ simultaneously. Without loss of generality, at each iteration, we render Gaussians from $\mathcal{G}_{0}$ at both $t=0$ and $t=1$ (where Gaussians are appropriately transformed into $t=1$ before rendering). Let $\hat{\boldsymbol{I}}_{0}$ and $\hat{\boldsymbol{I}}_{1}$ represent the set of rendered images at timestep $t_{0}$ and $t_{1}$ respectively. Similarly, let $\boldsymbol{I}_{0}$ and $\boldsymbol{I}_{1}$ represent the set of ground truth images at timestep $t_{0}$ and $t_{1}$ respectively. We use the following RGB loss that combines L1 and LPIPS [32]:

[TABLE]

where $\beta$ is a weight balancing the rendering contributions from each state. By backpropagating this loss, the parameters of both the Gaussian models and the unary potential fields are refined, thereby improving rendering quality and transformation coherence.

Dropout.

Notice that a trivial solution to Eq 2 occurs when all Gaussians are mapped to the origin in the canonical space, collapsing to a single point to achieve minimal positional difference. This can happen when the potential fields simply make the outputs satisfy $\boldsymbol{R}\boldsymbol{\mu}=-\boldsymbol{t}$. To avoid this trivial solution, random dropout is applied to the position input $\boldsymbol{\mu}$ during the training.

3.4 Interpolation and Extrapolation

Eq. 6 provides relative rotation $\boldsymbol{R}_{\text{r}}=\left(\boldsymbol{R}_{j}^{{(1)}}\right)^{\top}\boldsymbol{R}_{i}^{(0)}$ and relative translation $\boldsymbol{t}_{\text{r}}=\left(\boldsymbol{R}_{j}^{{(1)}}\right)^{\top}\left(\boldsymbol{t}_{i}^{0}-\boldsymbol{t}_{j}^{1}\right)$. Interpolation between $t=0$ and $t=1$ proceeds by interpolating between the identity transformation and this relative transformation. Note that our method is flexible with the interpolation strategy, resulting in diverse motion speeds, and we will demonstrate simple linear interpolation. Specifically, for interpolation parameter $t\in[0,1]$, the interpolated rotation $\boldsymbol{R}_{t}$ is computed using spherical linear interpolation (SLERP) [22] between the identity matrix and $\boldsymbol{R}_{\text{r}}$, while the interpolated translation $\boldsymbol{t}_{t}$ employs linear interpolation between zero and $\boldsymbol{t}_{\text{r}}$. For extrapolation, we simply set parameter $t<0$ or $t>1$, extending motion beyond observed timesteps.

4 Experiments

4.1 Dataset

We evaluate our method on both synthetic and real-world scenes, including single-object and multi-object scenes with local or global motion. Our evaluation dataset comprises: (1) the local-motion dataset from PAPR [20] containing six synthetic scenes and two real-world scenes, (2) real-world scenes from Dynamic Gaussian [17] with frame gaps of 5 timesteps to generate large-motion scenarios, (3) eight synthetic scenes created using the Objaverse objects [7], including three single-object scenes with large global motion and pose changes, and five multi-object scenes with global motion exhibiting rigid or non-rigid behavior, and (4) three manually captured real-world scenes, including one single-object and two multi-object scenes with global motion.

4.2 Motion Interpolation

Implementation Details.

For the 3DGS pre-training, we build upon Feature 3DGS [36] codebase and incorporate gsplat [29] for accelerated rasterization. For each state, we train for 30k iterations. Before the MLPs training, normalization scalars for the DINO, PCA-DINO, RGB, and Gaussian means are pre-calculated from the first 3DGS model. These normalizations are applied to both models during the query and database construction. The weights are set as $w_{c}=1$, $w_{f}=10$, $w_{\mu}=10$, $k=256$, $\beta=1$, with $\alpha$ starting at [math] and linearly increasing to $10$ over 10k iterations. The four MLPs are trained for 20k iterations, followed by 20k joint refinement iterations with enabled pruning and densification for both 3DGS models. More details can be found in the supplementary material.

Scene Interpolation Metrics.

When objects undergo large motions, determining the motion trajectories is an ill-posed problem, as there can be infinitely many paths for an object to transit from one state to another. Since we do not have access to a single “ground truth” trajectory, we cannot define metrics that measure the correctness of the predicted trajectory. One way to evaluate the trajectory is by the smoothness of interpolation. PAPR in Motion [20] proposed various such metrics to measure the smoothness of the overall interpolation by using only start and end state information. Specifically, rendering quality is evaluated using Scene Interpolation Fréchet inception distance [10] (SI-FID), while geometry quality is assessed using Scene Interpolation Earth Mover’s Distance (SI-EMD). Furthermore, we propose using Scene Interpolation Multiscale Potential Energy Discrepancy [27] (SI-MPED) to measure the local geometry preservation (details in supplementary material). To achieve a low SI-MPED value, a method needs to predict the correct end-state while maintaining good local geometry. Therefore, a high SI-MPED indicates either poor motion learning or loss of local geometry, or both. Note that SI-EMD and SI-MPED results are reported in units of $10^{-3}$ in this paper, and for all three metrics, lower scores indicate better quality.

Note about Baselines.

Methods that use deformation fields, namely 4DGS [25] and Deformable 3DGS [28], jointly optimize over the parameters of the deformation field and the Gaussian model. For large motion, this problem is ill-posed, and thus the training of these methods is very unstable. Hence, in tables and figures, we do not report results where the methods fail to train.

Quantitative Results.

Evaluation results of interpolation on synthetic scenes are reported in Table 1 and Table 2, and on real-world scenes in Table 3. For both synthetic and real-world scenes, our method outperforms the baselines by a significant margin, indicating better geometry and rendering quality during interpolation.

Qualitative Results.

Qualitative results are shown in Figure 2, Figure 7, Figure 3 and Figure 6. In Figure 2, our method is compared with the baseline methods PAPR in Motion [20] and Dynamic Gaussian [17] on novel-view synthesis during interpolation for synthetic scenes. Consider the Dolphin scene in Figure 2, where the baselines fail because they update the point positions locally and are not able to correspond to true positions that are far away. Figure 7 shows the results of novel-view synthesis during interpolation of the real-world scene Box. While the interpolation given by the baseline methods is suboptimal, our method correctly identifies such complex motion and produces feasible interpolation of the lid while moving the box body. Furthermore, Figure 3 shows that our method is able to produce plausible interpolation in real-world scenes when the bat has large motion between start and end states, while the bat disappears during the interpolation for Dynamic Gaussian [17]. Figure 6 presents qualitative results on both interpolation and extrapolation. Since neither baseline method can perform extrapolation, we focus on our method, demonstrating that it performs well on synthetic and real-world data, single and multiple objects, and both local and global motion.

4.3 Ablation Study

Each component is removed separately from the full model, and the average resulting metrics are reported in Table 4. Specifically, we study the importance of (1) the DINO input to the Motion Network, (2) the position (Gaussian mean) input to the Motion Network, (3) local distance preservation loss, and (4) the appearance refinement stage. Table 4 shows that the full model achieves the best SI-FID, SI-EMD, and SI-MPED.

5 Conclusion

We present Global Motion Corresponder (GMC), a novel method for 3D scene interpolation and extrapolation. We show that predicting large motion between timesteps is analogous to establishing smooth global correspondences between points across time. Next, we present our methods with the desired interpolation properties. Experimental results demonstrate that GMC significantly outperforms prior work in interpolation tasks, especially when dealing with large changes between captures. Moreover, GMC enables extrapolation beyond the captured states, a capability lacking in prior work.

A Implementation Details

The input positions and PCA-DINO for the MLPs are normalized using scalars pre-calculated from the start-state 3DGS model. The position input is then scaled by a hyperparameter weight, selected from $\{0.1,1.0\}$ based on the importance of the positional information. Correspondingly, dropout is applied to the position input to avoid trivial local minima, with a ratio of $0.1$ or $0.2$ depending on the previously chosen scale. To mitigate the issue of getting trapped in local minima, the Perturb-and-MAP strategy [19] is applied to the total energy, where the perturbations are sampled from a Gumbel distribution. The learning rate of training the MLPs is set to $0.0005$, using the Adam optimizer [13] with default parameters. For the RGB loss during the joint refinement, L1 and LPIPS [32] losses are combined with weights $1.0$ and $0.1$, respectively; gradients from the MLPs are not used to update the 3DGS models. Due to the large size of Gaussian sets, batches of Gaussians are sampled during each iteration when searching for the minimum energy between the two Gaussian sets. For most scenes, the batch size is set to $20,000$, and it can be reduced accordingly if the total number of Gaussians is smaller. The FAISS library [11] is used to perform efficient nearest neighbor searches.

B SI-MPED Metric

For each interpolation step, the Multiscale Potential Energy Discrepancy (MPED) [27] is calculated between the interpolated point cloud and the ground-truth point cloud from the start and end states, respectively. The MPED is computed by aggregating distances from the neighborhoods comprising $0.1\%$, $0.5\%$, and $1\%$ of the total points, summing these values to obtain the overall MEPD. Following PAPR in Motion [20], the Scene Interpolation MPED (SI-MPED) is defined as a weighted sum of the MEPD at each interpolation step, where the weights are proportional to the average distance movement of points compared to the total movement from the start to the end states.

C Motion Interpolation and Extrapolation

Qualitative results on motion interpolation and extrapolation for global-motion scenes are presented in Figure 10. Additionally, qualitative results on local-motion scenes [20] are shown in Figure 11, and the quantitative interpolation evaluations for these scenes are provided in Table 7. Qualitative interpolation results on two real-world scenes from Dynamic Gaussian [17] are provided in Figure 9, in comparison with the Dynamic Gaussian [17] baseline.

D Ablation Study

We find the following properties when removing one of the key components of our method:

Removing DINO input can result in implausible interpolation (Ball), wrong global motion interpolation (Boat), or wrong local motion interpolation. 2. 2.

Removing position input can result in wrong global matching (Ball and Car) or wrong local motion interpolation (Butterfly). 3. 3.

Removing local isometry loss can result in noisy floaters (Dolphin) or blurry rendering (Butterfly and Microwave) during the interpolation. 4. 4.

Removing local isometry loss can result in noisy rendering (Ball and Microwave) during the interpolation or suboptimal end status prediction (Butterfly).

E Sparse View Refinement

In addition to motion interpolation and extrapolation, GMC can also be used to improve reconstruction quality in sparse capture scenarios. Specifically, only five or ten views are available for sparse captures, and we consider two settings: (1) the start state has dense views and the end state has sparse views, and (2) both states have sparse views. When the input views are sparse, the reconstructed 3DGS will have bad geometry and thus will perform poorly in novel view synthesis. While a single state might not have enough views for good 3D reconstruction, we can borrow the information from the other state so that it can refine the self geometry and thus improve the novel view synthesis. Specifically, through the rendering loss $\mathcal{L}_{\mathrm{RGB}}(\boldsymbol{I}_{f},\hat{\boldsymbol{I}}_{f})$ in Eq. 9, the sparse-view 3DGS can use the training views from the other state, and thus improve itself.

Results.

For the sparse-view setting, we set $\beta=5$, because in this setting, the ground-truth training views are more reliable than the ”borrowed” information based on Gaussian matching. Otherwise, each scene has the same setting as the dense-dense setting studied for interpolation and extrapolation tasks. We showcase three examples of the application in sparse-view refinement in Figure 8, and we report quantitative results, average PSNR, SSIM [24], and LPIPS [32] of novel-view synthesis of 200 views. Quantitative results on sparse-view refinement with the sparse-sparse view setting are presented in Table 6, and results with the sparse-dense view setting are shown in Table 5.

Both qualitative and quantitative results show that our method significantly improves upon the vanilla 3DGS trained on sparse views. When training views are few and sparse, two significant issues arise: (1) the presence of floaters, and (2) a lack of details in under-observed regions. The qualitative results show that our method is able to reduce (1) and handle (2). Our method reduces floaters because they have a poor match in the other state, and thus, when transformed and rendered, they can be removed by the rendering loss. Our method improves details in under-observed regions because the other state may have more information on appearance details, which can be borrowed to enhance the current state.