Self-Supervised Generative-Contrastive Learning of Multi-Modal Euclidean Input for 3D Shape Latent Representations: A Dynamic Switching Approach

TL;DR

This paper introduces a self-supervised learning framework combining generative and contrastive methods with a dynamic switching approach to learn effective 3D shape representations from multi-modal data, improving reconstruction and classification.

Contribution

It presents a novel dynamic switching training strategy for multi-modal 3D shape encoding that prevents collapse and enhances feature integration from different data modalities.

Findings

Improved 3D shape reconstruction accuracy.

Enhanced classification performance using learned latent representations.

Effective multi-modal data integration in self-supervised learning.

Abstract

We propose a combined generative and contrastive neural architecture for learning latent representations of 3D volumetric shapes. The architecture uses two encoder branches for voxel grids and multi-view images from the same underlying shape. The main idea is to combine a contrastive loss between the resulting latent representations with an additional reconstruction loss. That helps to avoid collapsing the latent representations as a trivial solution for minimizing the contrastive loss. A novel dynamic switching approach is used to cross-train two encoders with a shared decoder. The switching approach also enables the stop gradient operation on a random branch. Further classification experiments show that the latent representations learned with our self-supervised method integrate more useful information from the additional input data implicitly, thus leading to better reconstruction and classification performance.