Efficient 3D Recognition with Event-driven Spike Sparse Convolution

TL;DR

This paper introduces a novel energy-efficient 3D SNN backbone using Spike Voxel Coding and Spike Sparse Convolution, achieving state-of-the-art accuracy on 3D recognition tasks while being hardware-friendly.

Contribution

The paper proposes the first direct training 3D SNN backbone that effectively processes point clouds for multiple 3D vision tasks, improving efficiency and accuracy.

Findings

Achieved 91.7% top-1 accuracy on ModelNet40 with 1.87M parameters.

Outperformed larger SNN baselines with 14.3M parameters.

Demonstrated state-of-the-art results on ModelNet40, KITTI, and Semantic KITTI datasets.

Abstract

Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. Point clouds are sparse 3D spatial data, which suggests that SNNs should be well-suited for processing them. However, when applying SNNs to point clouds, they often exhibit limited performance and fewer application scenarios. We attribute this to inappropriate preprocessing and feature extraction methods. To address this issue, we first introduce the Spike Voxel Coding (SVC) scheme, which encodes the 3D point clouds into a sparse spike train space, reducing the storage requirements and saving time on point cloud preprocessing. Then, we propose a Spike Sparse Convolution (SSC) model for efficiently extracting 3D sparse point cloud features. Combining SVC and SSC, we design an efficient 3D SNN backbone (E-3DSNN), which is friendly with neuromorphic hardware. For instance, SSC can be implemented on neuromorphic chips with only minor modifications to the addressing function of vanilla spike convolution. Experiments on ModelNet40, KITTI, and Semantic KITTI datasets demonstrate that E-3DSNN achieves state-of-the-art (SOTA) results with remarkable efficiency. Notably, our E-3DSNN (1.87M) obtained 91.7\% top-1 accuracy on ModelNet40, surpassing the current best SNN baselines (14.3M) by 3.0\%. To our best knowledge, it is the first direct training 3D SNN backbone that can simultaneously handle various 3D computer vision tasks (e.g., classification, detection, and segmentation) with an event-driven nature. Code is available: https://github.com/bollossom/E-3DSNN/.