Pointy - A Lightweight Transformer for Point Cloud Foundation Models

TL;DR

Pointy introduces a lightweight transformer architecture for point cloud data that achieves competitive results with less training data and complexity, emphasizing the importance of architecture and training setup.

Contribution

The paper presents a novel, efficient transformer-based point cloud model trained on fewer samples that outperforms larger models, with a comprehensive benchmarking framework.

Findings

Our model outperforms larger models trained on more data.

Simple backbones can achieve competitive results.

Standardized benchmarks highlight architecture benefits.

Abstract

Foundation models for point cloud data have recently grown in capability, often leveraging extensive representation learning from language or vision. In this work, we take a more controlled approach by introducing a lightweight transformer-based point cloud architecture. In contrast to the heavy reliance on cross-modal supervision, our model is trained only on 39k point clouds - yet it outperforms several larger foundation models trained on over 200k training samples. Interestingly, our method approaches state-of-the-art results from models that have seen over a million point clouds, images, and text samples, demonstrating the value of a carefully curated training setup and architecture. To ensure rigorous evaluation, we conduct a comprehensive replication study that standardizes the training regime and benchmarks across multiple point cloud architectures. This unified experimental framework isolates the impact of architectural choices, allowing for transparent comparisons and highlighting the benefits of our design and other tokenizer-free architectures. Our results show that simple backbones can deliver competitive results to more complex or data-rich strategies. The implementation, including code, pre-trained models, and training protocols, is available at https://github.com/KonradSzafer/Pointy.