Sampled Transformer for Point Sets

TL;DR

This paper introduces a sampled transformer model that efficiently processes point sets with $O(n)$ complexity, using random sampling and shared attention to approximate dense attention, achieving high accuracy in point-cloud tasks.

Contribution

The paper proposes a novel $O(n)$ sampled transformer for point sets that employs random element sampling and shared Hamiltonian attention, enabling efficient and universal set-to-set function approximation.

Findings

Achieves comparable or better accuracy than dense transformers on point-cloud tasks.

Reduces computational complexity from $O(n^2)$ to $O(n)$ in attention mechanisms.

Demonstrates universal approximation capability for continuous set-to-set functions.

Abstract

The sparse transformer can reduce the computational complexity of the self-attention layers to $O(n)$, whilst still being a universal approximator of continuous sequence-to-sequence functions. However, this permutation variant operation is not appropriate for direct application to sets. In this paper, we proposed an $O(n)$ complexity sampled transformer that can process point set elements directly without any additional inductive bias. Our sampled transformer introduces random element sampling, which randomly splits point sets into subsets, followed by applying a shared Hamiltonian self-attention mechanism to each subset. The overall attention mechanism can be viewed as a Hamiltonian cycle in the complete attention graph, and the permutation of point set elements is equivalent to randomly sampling Hamiltonian cycles. This mechanism implements a Monte Carlo simulation of the $O(n^2)$ dense attention connections. We show that it is a universal approximator for continuous set-to-set functions. Experimental results on point-clouds show comparable or better accuracy with significantly reduced computational complexity compared to the dense transformer or alternative sparse attention schemes.