Image Neural Field Diffusion Models

TL;DR

This paper introduces diffusion models trained on image neural fields, enabling continuous, resolution-independent image generation and improved performance over fixed-resolution models, with applications in inverse problems and mixed-resolution datasets.

Contribution

It presents a novel approach to learn continuous image distributions via neural fields, enhancing diffusion models' flexibility and effectiveness over traditional fixed-resolution methods.

Findings

Outperforms fixed-resolution diffusion models with super-resolution.

Enables training on mixed-resolution datasets.

Efficiently solves inverse problems with multi-scale conditions.

Abstract

Diffusion models have shown an impressive ability to model complex data distributions, with several key advantages over GANs, such as stable training, better coverage of the training distribution's modes, and the ability to solve inverse problems without extra training. However, most diffusion models learn the distribution of fixed-resolution images. We propose to learn the distribution of continuous images by training diffusion models on image neural fields, which can be rendered at any resolution, and show its advantages over fixed-resolution models. To achieve this, a key challenge is to obtain a latent space that represents photorealistic image neural fields. We propose a simple and effective method, inspired by several recent techniques but with key changes to make the image neural fields photorealistic. Our method can be used to convert existing latent diffusion autoencoders into image neural field autoencoders. We show that image neural field diffusion models can be trained using mixed-resolution image datasets, outperform fixed-resolution diffusion models followed by super-resolution models, and can solve inverse problems with conditions applied at different scales efficiently.