PointConvFormer: Revenge of the Point-based Convolution

TL;DR

PointConvFormer is a new point cloud neural network component that combines point convolution and attention mechanisms, improving accuracy and efficiency in 3D scene understanding tasks like segmentation and scene flow estimation.

Contribution

It introduces PointConvFormer, a novel building block that integrates feature-based attention with point convolution, enhancing neighborhood selection and invariance properties.

Findings

Outperforms classic convolution, transformer, and voxel-based methods in accuracy-speed tradeoff.

Effectively captures details at object boundaries and flat regions.

Demonstrates versatility across multiple 3D understanding tasks and datasets.

Abstract

We introduce PointConvFormer, a novel building block for point cloud based deep network architectures. Inspired by generalization theory, PointConvFormer combines ideas from point convolution, where filter weights are only based on relative position, and Transformers which utilize feature-based attention. In PointConvFormer, attention computed from feature difference between points in the neighborhood is used to modify the convolutional weights at each point. Hence, we preserved the invariances from point convolution, whereas attention helps to select relevant points in the neighborhood for convolution. PointConvFormer is suitable for multiple tasks that require details at the point level, such as segmentation and scene flow estimation tasks. We experiment on both tasks with multiple datasets including ScanNet, SemanticKitti, FlyingThings3D and KITTI. Our results show that PointConvFormer offers a better accuracy-speed tradeoff than classic convolutions, regular transformers, and voxelized sparse convolution approaches. Visualizations show that PointConvFormer performs similarly to convolution on flat areas, whereas the neighborhood selection effect is stronger on object boundaries, showing that it has got the best of both worlds.