Frequency-domain Learning for Volumetric-based 3D Data Perception

TL;DR

This paper explores frequency-domain learning for 3D volumetric data perception, demonstrating significant data size reduction with minimal accuracy loss and revealing spectral bias in 3D CNNs.

Contribution

It is the first study to analyze frequency-domain learning in 3D CNNs, showing its effectiveness in reducing input data size while maintaining accuracy.

Findings

Reduces 3D input data size by 98% with less than 2% accuracy drop.

Improves segmentation accuracy by 1.48% with 98% data reduction.

Enhances accuracy and IoU by over 3% when using higher-resolution data.

Abstract

Frequency-domain learning draws attention due to its superior tradeoff between inference accuracy and input data size. Frequency-domain learning in 2D computer vision tasks has shown that 2D convolutional neural networks (CNN) have a stationary spectral bias towards low-frequency channels so that high-frequency channels can be pruned with no or little accuracy degradation. However, frequency-domain learning has not been studied in the context of 3D CNNs with 3D volumetric data. In this paper, we study frequency-domain learning for volumetric-based 3D data perception to reveal the spectral bias and the accuracy-input-data-size tradeoff of 3D CNNs. Our study finds that 3D CNNs are sensitive to a limited number of critical frequency channels, especially low-frequency channels. Experiment results show that frequency-domain learning can significantly reduce the size of volumetric-based 3D inputs (based on spectral bias) while achieving comparable accuracy with conventional spatial-domain learning approaches. Specifically, frequency-domain learning is able to reduce the input data size by 98% in 3D shape classification while limiting the average accuracy drop within 2%, and by 98% in the 3D point cloud semantic segmentation with a 1.48% mean-class accuracy improvement while limiting the mean-class IoU loss within 1.55%. Moreover, by learning from higher-resolution 3D data (i.e., 2x of the original image in the spatial domain), frequency-domain learning improves the mean-class accuracy and mean-class IoU by 3.04% and 0.63%, respectively, while achieving an 87.5% input data size reduction in 3D point cloud semantic segmentation.