Benefit from Reference: Retrieval-Augmented Cross-modal Point Cloud Completion

TL;DR

This paper introduces a retrieval-augmented framework for 3D point cloud completion that leverages cross-modal retrieval and hierarchical feature fusion to improve structural accuracy and generalization to sparse and unseen data.

Contribution

It proposes a novel retrieval-augmented completion framework with a Structural Shared Feature Encoder and Progressive Retrieval-Augmented Generator for enhanced 3D point cloud reconstruction.

Findings

Effective in generating fine-grained point clouds

Demonstrates strong generalization to sparse data

Handles unseen categories well

Abstract

Completing the whole 3D structure based on an incomplete point cloud is a challenging task, particularly when the residual point cloud lacks typical structural characteristics. Recent methods based on cross-modal learning attempt to introduce instance images to aid the structure feature learning. However, they still focus on each particular input class, limiting their generation abilities. In this work, we propose a novel retrieval-augmented point cloud completion framework. The core idea is to incorporate cross-modal retrieval into completion task to learn structural prior information from similar reference samples. Specifically, we design a Structural Shared Feature Encoder (SSFE) to jointly extract cross-modal features and reconstruct reference features as priors. Benefiting from a dual-channel control gate in the encoder, relevant structural features in the reference sample are enhanced and irrelevant information interference is suppressed. In addition, we propose a Progressive Retrieval-Augmented Generator (PRAG) that employs a hierarchical feature fusion mechanism to integrate reference prior information with input features from global to local. Through extensive evaluations on multiple datasets and real-world scenes, our method shows its effectiveness in generating fine-grained point clouds, as well as its generalization capability in handling sparse data and unseen categories.