Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction

TL;DR

This paper introduces a spectral diagnostic framework to evaluate feature upsamplers in 2D-to-3D scene reconstruction, revealing that spectral consistency, not spatial sharpness, is key for better 3D reconstruction quality.

Contribution

The paper proposes a novel spectral diagnostic framework with six metrics to analyze the impact of upsampling methods on 3D reconstruction, emphasizing spectral consistency over spatial detail.

Findings

Spectral structural consistency strongly predicts NVS quality.

High-frequency spectral slope drift negatively correlates with reconstruction performance.

Classical interpolation often outperforms learnable upsamplers in reconstruction quality.

Abstract

A typical 2D-to-3D pipeline takes multi-view images as input, where a Vision Foundation Model (VFM) extracts features that are spatially upsampled to dense representations for 3D reconstruction. If dense features across views preserve geometric consistency, differentiable rendering can recover an accurate 3D representation, making the feature upsampler a critical component. Recent learnable upsampling methods mainly aim to enhance spatial details, such as sharper geometry or richer textures, yet their impact on 3D awareness remains underexplored. To address this gap, we introduce a spectral diagnostic framework with six complementary metrics that characterize amplitude redistribution, structural spectral alignment, and directional stability. Across classical interpolation and learnable upsampling methods on CLIP and DINO backbones, we observe three key findings. First, structural spectral consistency (SSC/CSC) is the strongest predictor of NVS quality, whereas High-Frequency Spectral Slope Drift (HFSS) often correlates negatively with reconstruction performance, indicating that emphasizing high-frequency details alone does not necessarily improve 3D reconstruction. Second, geometry and texture respond to different spectral properties: Angular Energy Consistency (ADC) correlates more strongly with geometry-related metrics, while SSC/CSC influence texture fidelity slightly more than geometric accuracy. Third, although learnable upsamplers often produce sharper spatial features, they rarely outperform classical interpolation in reconstruction quality, and their effectiveness depends on the reconstruction model. Overall, our results indicate that reconstruction quality is more closely related to preserving spectral structure than to enhancing spatial detail, highlighting spectral consistency as an important principle for designing upsampling strategies in 2D-to-3D pipelines.