MATE: Masked Autoencoders are Online 3D Test-Time Learners

TL;DR

MATE introduces a novel test-time training method for 3D point cloud classification that enhances robustness to distribution shifts by using masked autoencoding with minimal test data adaptation.

Contribution

It is the first TTT approach for 3D data that employs masked autoencoders for efficient, real-time test-time adaptation of deep networks.

Findings

Significantly improves robustness to 3D data corruptions

Effective with as little as 5% of test points for adaptation

Achieves competitive performance with low computational overhead

Abstract

Our MATE is the first Test-Time-Training (TTT) method designed for 3D data, which makes deep networks trained for point cloud classification robust to distribution shifts occurring in test data. Like existing TTT methods from the 2D image domain, MATE also leverages test data for adaptation. Its test-time objective is that of a Masked Autoencoder: a large portion of each test point cloud is removed before it is fed to the network, tasked with reconstructing the full point cloud. Once the network is updated, it is used to classify the point cloud. We test MATE on several 3D object classification datasets and show that it significantly improves robustness of deep networks to several types of corruptions commonly occurring in 3D point clouds. We show that MATE is very efficient in terms of the fraction of points it needs for the adaptation. It can effectively adapt given as few as 5% of tokens of each test sample, making it extremely lightweight. Our experiments show that MATE also achieves competitive performance by adapting sparsely on the test data, which further reduces its computational overhead, making it ideal for real-time applications.