PoI: A Filter to Extract Pixel of Interest from Novel Views for Scene Coordinate Regression

TL;DR

This paper introduces PoI, a framework that enhances scene coordinate regression for visual localization by refining novel view synthesis with diffusion models and a pixel filtering strategy, improving accuracy and robustness.

Contribution

It presents a novel approach combining diffusion-based view refinement and pixel filtering to improve SCR training with synthetic views, addressing limitations of existing NVS methods.

Findings

Consistently improves localization accuracy on 7Scenes and Cambridge Landmarks.

Achieves state-of-the-art performance with efficient training.

Demonstrates importance of pixel reliability control in synthetic view augmentation.

Abstract

Neural View Synthesis (NVS) techniques such as NeRF and 3D Gaussian Splatting (3DGS) have enabled photorealistic rendering from novel viewpoints and are increasingly used to augment training data for visual localization. However, these methods fundamentally rely on observed geometry and radiance; they interpolate existing information but cannot hallucinate unseen 3D structures or recover missing content under sparse or extreme viewpoints. As a result, rendered views often exhibit blur, structural distortion, or incomplete geometry. While such imperfections may be tolerated by Camera Pose Regression (CPR) methods, they severely degrade Scene Coordinate Regression (SCR), which requires accurate per-pixel 3D supervision. To address this limitation, we introduce PoI (Pixel-of-Interest), a framework that enables effective NVS augmentation for SCR-based localization. We first employ 3DGS to render novel views and leverage a single-step diffusion model to refine them, allowing the synthesis of structurally plausible details beyond purely geometry-driven interpolation. However, even diffusion-refined views may contain unreliable pixels. Therefore, we propose a progressive pixel-level filtering strategy based on reprojection error to selectively retain trustworthy synthetic pixels during training while suppressing harmful ones. Extensive experiments on 7Scenes and Cambridge Landmarks demonstrate that our method consistently improves localization accuracy over strong SCR baselines and achieves state-of-the-art performance with competitive training efficiency. Our results reveal that, for SCR, the benefit of novel view augmentation depends not only on generative realism but also on explicit control of pixel-level reliability.