Efficient Dense Matching for Enhanced Gaussian Splatting Using AV1 Motion Vectors

TL;DR

This paper introduces an AV1-based feature matching pipeline that enhances initial point cloud density and reduces processing time for 3D Gaussian Splatting, leading to improved scene reconstruction quality.

Contribution

It leverages AV1 motion vectors for efficient feature matching, significantly densifies point clouds, and improves 3DGS reconstruction performance and speed.

Findings

Up to eight times more points in initial point clouds.

9-point increase in VMAF for reconstruction quality.

63% reduction in training time to reach baseline quality.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a prominent framework for real-time, photorealistic scene reconstruction, offering significant speed-ups over Neural Radiance Fields (NeRF). However, the fidelity of 3DGS representations remains heavily dependent on the quality of the initial point cloud. While standard Structure-from-Motion (SfM) pipelines using COLMAP provide adequate initialisation, they often suffer from high computational costs and sparsity in textureless regions, which degrades subsequent reconstruction accuracy and convergence speed. In this work, we introduce an AV1-based feature detection and matching pipeline that significantly reduces SfM processing overhead. By leveraging motion vectors inherent to the AV1 video codec, we bypass computationally expensive exhaustive matching while maintaining geometric robustness. Our pipeline produces substantially denser point clouds, with up to eight times as many points as classical SfM. We demonstrate that this enhanced initialisation directly improves 3DGS performance, yielding an 9-point increase in VMAF and a 63% average reduction in training time required to reach baseline quality. The project page: https://sigmedia.tv/AV1-3DGS.github.io/