Beyond Flatland: Pre-training with a Strong 3D Inductive Bias

TL;DR

This paper explores pre-training models using synthetic 3D object renders with procedural generation, aiming to improve transfer learning and understand biological vision by comparing 2D fractals and 3D objects.

Contribution

It introduces a novel approach of using 3D procedural object renders for pre-training, extending prior work on synthetic fractals, and investigates their impact on transfer learning and biological vision modeling.

Findings

Pre-training with 3D renders enhances transfer learning performance.

Varying illumination and pose improves model invariance to 3D transformations.

Comparison with brain data offers insights into 2D vs. 3D visual processing.

Abstract

Pre-training on large-scale databases consisting of natural images and then fine-tuning them to fit the application at hand, or transfer-learning, is a popular strategy in computer vision. However, Kataoka et al., 2020 introduced a technique to eliminate the need for natural images in supervised deep learning by proposing a novel synthetic, formula-based method to generate 2D fractals as training corpus. Using one synthetically generated fractal for each class, they achieved transfer learning results comparable to models pre-trained on natural images. In this project, we take inspiration from their work and build on this idea -- using 3D procedural object renders. Since the image formation process in the natural world is based on its 3D structure, we expect pre-training with 3D mesh renders to provide an implicit bias leading to better generalization capabilities in a transfer learning setting and that invariances to 3D rotation and illumination are easier to be learned based on 3D data. Similar to the previous work, our training corpus will be fully synthetic and derived from simple procedural strategies; we will go beyond classic data augmentation and also vary illumination and pose which are controllable in our setting and study their effect on transfer learning capabilities in context to prior work. In addition, we will compare the 2D fractal and 3D procedural object networks to human and non-human primate brain data to learn more about the 2D vs. 3D nature of biological vision.