Sampled Transformer for Point Sets
Shidi Li, Christian Walder, Alexander Soen, Lexing Xie, Miaomiao Liu

TL;DR
This paper introduces a sampled transformer model that efficiently processes point sets with $O(n)$ complexity, using random sampling and shared attention to approximate dense attention, achieving high accuracy in point-cloud tasks.
Contribution
The paper proposes a novel $O(n)$ sampled transformer for point sets that employs random element sampling and shared Hamiltonian attention, enabling efficient and universal set-to-set function approximation.
Findings
Achieves comparable or better accuracy than dense transformers on point-cloud tasks.
Reduces computational complexity from $O(n^2)$ to $O(n)$ in attention mechanisms.
Demonstrates universal approximation capability for continuous set-to-set functions.
Abstract
The sparse transformer can reduce the computational complexity of the self-attention layers to , whilst still being a universal approximator of continuous sequence-to-sequence functions. However, this permutation variant operation is not appropriate for direct application to sets. In this paper, we proposed an complexity sampled transformer that can process point set elements directly without any additional inductive bias. Our sampled transformer introduces random element sampling, which randomly splits point sets into subsets, followed by applying a shared Hamiltonian self-attention mechanism to each subset. The overall attention mechanism can be viewed as a Hamiltonian cycle in the complete attention graph, and the permutation of point set elements is equivalent to randomly sampling Hamiltonian cycles. This mechanism implements a Monte Carlo simulation of the …
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Advanced Image and Video Retrieval Techniques · Visual Attention and Saliency Detection
MethodsAttention Is All You Need · Cosine Annealing · Layer Normalization · Residual Connection · Dense Connections · Linear Layer · Dropout · Weight Decay · Multi-Head Attention · Refunds@Expedia|||How do I get a full refund from Expedia?
