Accelerating 3D Deep Learning with PyTorch3D

TL;DR

PyTorch3D is a modular, efficient, and differentiable library that accelerates 3D deep learning by providing scalable rendering and processing tools, enabling faster research and development in 3D applications.

Contribution

The paper introduces PyTorch3D, a new library that offers modular, efficient, and differentiable 3D operators and rendering, addressing engineering challenges in 3D deep learning.

Findings

PyTorch3D outperforms existing renderers in speed and memory efficiency.

It enables state-of-the-art unsupervised 3D shape prediction from 2D images.

The library is open-source to facilitate widespread adoption.

Abstract

Deep learning has significantly improved 2D image recognition. Extending into 3D may advance many new applications including autonomous vehicles, virtual and augmented reality, authoring 3D content, and even improving 2D recognition. However despite growing interest, 3D deep learning remains relatively underexplored. We believe that some of this disparity is due to the engineering challenges involved in 3D deep learning, such as efficiently processing heterogeneous data and reframing graphics operations to be differentiable. We address these challenges by introducing PyTorch3D, a library of modular, efficient, and differentiable operators for 3D deep learning. It includes a fast, modular differentiable renderer for meshes and point clouds, enabling analysis-by-synthesis approaches. Compared with other differentiable renderers, PyTorch3D is more modular and efficient, allowing users to more easily extend it while also gracefully scaling to large meshes and images. We compare the PyTorch3D operators and renderer with other implementations and demonstrate significant speed and memory improvements. We also use PyTorch3D to improve the state-of-the-art for unsupervised 3D mesh and point cloud prediction from 2D images on ShapeNet. PyTorch3D is open-source and we hope it will help accelerate research in 3D deep learning.