ProxyTransformation: Preshaping Point Cloud Manifold With Proxy Attention For 3D Visual Grounding

TL;DR

This paper introduces Proxy Transformation, a novel method that enhances point cloud manifolds for 3D visual grounding by using multimodal proxies, leading to state-of-the-art performance and reduced computational costs.

Contribution

The paper presents Proxy Transformation with Proxy Attention and submanifold transformation modules, improving point cloud quality for 3D grounding in real-time applications.

Findings

Achieves 7.49% improvement on easy targets

Achieves 4.60% improvement on hard targets

Reduces attention block computation by 40.6%

Abstract

Embodied intelligence requires agents to interact with 3D environments in real time based on language instructions. A foundational task in this domain is ego-centric 3D visual grounding. However, the point clouds rendered from RGB-D images retain a large amount of redundant background data and inherent noise, both of which can interfere with the manifold structure of the target regions. Existing point cloud enhancement methods often require a tedious process to improve the manifold, which is not suitable for real-time tasks. We propose Proxy Transformation suitable for multimodal task to efficiently improve the point cloud manifold. Our method first leverages Deformable Point Clustering to identify the point cloud sub-manifolds in target regions. Then, we propose a Proxy Attention module that utilizes multimodal proxies to guide point cloud transformation. Built upon Proxy Attention, we design a submanifold transformation generation module where textual information globally guides translation vectors for different submanifolds, optimizing relative spatial relationships of target regions. Simultaneously, image information guides linear transformations within each submanifold, refining the local point cloud manifold of target regions. Extensive experiments demonstrate that Proxy Transformation significantly outperforms all existing methods, achieving an impressive improvement of 7.49% on easy targets and 4.60% on hard targets, while reducing the computational overhead of attention blocks by 40.6%. These results establish a new SOTA in ego-centric 3D visual grounding, showcasing the effectiveness and robustness of our approach.