JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling

TL;DR

JointNet is a new neural network architecture that extends pre-trained text-to-image diffusion models to jointly model images and dense modalities like depth maps, enabling diverse applications with efficient training.

Contribution

It introduces a dense modality extension to pre-trained diffusion models by creating a parallel branch that is densely connected to the RGB branch, while keeping the original RGB branch fixed.

Findings

Effective joint RGBD generation demonstrated

High-quality dense depth prediction achieved

Versatile applications including depth-conditioned image generation and 3D panorama synthesis

Abstract

We introduce JointNet, a novel neural network architecture for modeling the joint distribution of images and an additional dense modality (e.g., depth maps). JointNet is extended from a pre-trained text-to-image diffusion model, where a copy of the original network is created for the new dense modality branch and is densely connected with the RGB branch. The RGB branch is locked during network fine-tuning, which enables efficient learning of the new modality distribution while maintaining the strong generalization ability of the large-scale pre-trained diffusion model. We demonstrate the effectiveness of JointNet by using RGBD diffusion as an example and through extensive experiments, showcasing its applicability in a variety of applications, including joint RGBD generation, dense depth prediction, depth-conditioned image generation, and coherent tile-based 3D panorama generation.