Permutation Matters: Anisotropic Convolutional Layer for Learning on Point Clouds

TL;DR

This paper introduces PAI-Conv, a novel anisotropic convolutional layer for point clouds that uses soft permutation matrices and attention mechanisms, improving representation power for 3D vision tasks.

Contribution

It proposes PAI-Conv, a permutable anisotropic convolution that leverages attention-based permutation matrices, bridging ideas from NLP transformers to point cloud learning.

Findings

Achieves competitive classification accuracy on benchmark datasets.

Improves semantic segmentation performance over existing methods.

Demonstrates robustness with random point sampling.

Abstract

It has witnessed a growing demand for efficient representation learning on point clouds in many 3D computer vision applications. Behind the success story of convolutional neural networks (CNNs) is that the data (e.g., images) are Euclidean structured. However, point clouds are irregular and unordered. Various point neural networks have been developed with isotropic filters or using weighting matrices to overcome the structure inconsistency on point clouds. However, isotropic filters or weighting matrices limit the representation power. In this paper, we propose a permutable anisotropic convolutional operation (PAI-Conv) that calculates soft-permutation matrices for each point using dot-product attention according to a set of evenly distributed kernel points on a sphere's surface and performs shared anisotropic filters. In fact, dot product with kernel points is by analogy with the dot-product with keys in Transformer as widely used in natural language processing (NLP). From this perspective, PAI-Conv can be regarded as the transformer for point clouds, which is physically meaningful and is robust to cooperate with the efficient random point sampling method. Comprehensive experiments on point clouds demonstrate that PAI-Conv produces competitive results in classification and semantic segmentation tasks compared to state-of-the-art methods.