Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence

TL;DR

This paper introduces a novel approach to point cloud matching by integrating fixed Gaussian-based self-attention weights into Transformer models, improving training efficiency and robustness to noise.

Contribution

It proposes using fixed Gaussian attention weights in Transformers for point cloud matching, reducing training time and increasing noise robustness.

Findings

Fixing attention weights accelerates training.

Enhanced stability and robustness to noisy data.

Layer-wise impact analysis of Gaussian attention.

Abstract

Current data-driven methodologies for point cloud matching demand extensive training time and computational resources, presenting significant challenges for model deployment and application. In the point cloud matching task, recent advancements with an encoder-only Transformer architecture have revealed the emergence of semantically meaningful patterns in the attention heads, particularly resembling Gaussian functions centered on each point of the input shape. In this work, we further investigate this phenomenon by integrating these patterns as fixed attention weights within the attention heads of the Transformer architecture. We evaluate two variants: one utilizing predetermined variance values for the Gaussians, and another where the variance values are treated as learnable parameters. Additionally we analyze the performances on noisy data and explore a possible way to improve robustness to noise. Our findings demonstrate that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization. Furthermore, we conducted an ablation study to identify the specific layers where the infused information is most impactful and to understand the reliance of the network on this information.