MetaSets: Meta-Learning on Point Sets for Generalizable Representations

TL;DR

MetaSets introduces a meta-learning approach for point cloud representations that generalize across unseen domains, addressing the challenge of geometry shifts from simulated to real data in 3D vision tasks.

Contribution

The paper proposes MetaSets, a meta-learning framework that enhances the generalizability of point cloud models to unseen domains without access during training.

Findings

MetaSets outperforms existing methods on new benchmarks.

MetaSets achieves significant improvements in sim-to-real transfer.

The approach effectively handles geometry shifts in 3D point clouds.

Abstract

Deep learning techniques for point clouds have achieved strong performance on a range of 3D vision tasks. However, it is costly to annotate large-scale point sets, making it critical to learn generalizable representations that can transfer well across different point sets. In this paper, we study a new problem of 3D Domain Generalization (3DDG) with the goal to generalize the model to other unseen domains of point clouds without any access to them in the training process. It is a challenging problem due to the substantial geometry shift from simulated to real data, such that most existing 3D models underperform due to overfitting the complete geometries in the source domain. We propose to tackle this problem via MetaSets, which meta-learns point cloud representations from a group of classification tasks on carefully-designed transformed point sets containing specific geometry priors. The learned representations are more generalizable to various unseen domains of different geometries. We design two benchmarks for Sim-to-Real transfer of 3D point clouds. Experimental results show that MetaSets outperforms existing 3D deep learning methods by large margins.