Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Images

TL;DR

This paper introduces a novel method for robust 3D scene graph generation from multi-view RGB images, overcoming noisy geometry and background noise by enriching features with semantic, spatial, and neighboring information, and leveraging statistical priors.

Contribution

It proposes a new approach that enhances feature robustness and accuracy in 3D scene graph estimation using only multi-view images, without relying on ground truth 3D annotations.

Findings

Outperforms existing methods using only multi-view images.

Effectively filters background noise through semantic masks.

Improves robustness by incorporating neighboring node information.

Abstract

Modern 3D semantic scene graph estimation methods utilize ground truth 3D annotations to accurately predict target objects, predicates, and relationships. In the absence of given 3D ground truth representations, we explore leveraging only multi-view RGB images to tackle this task. To attain robust features for accurate scene graph estimation, we must overcome the noisy reconstructed pseudo point-based geometry from predicted depth maps and reduce the amount of background noise present in multi-view image features. The key is to enrich node and edge features with accurate semantic and spatial information and through neighboring relations. We obtain semantic masks to guide feature aggregation to filter background features and design a novel method to incorporate neighboring node information to aid robustness of our scene graph estimates. Furthermore, we leverage on explicit statistical priors calculated from the training summary statistics to refine node and edge predictions based on their one-hop neighborhood. Our experiments show that our method outperforms current methods purely using multi-view images as the initial input. Our project page is available at https://qixun1.github.io/projects/SCRSSG.