Memory Efficient Patch-based Training for INR-based GANs

TL;DR

This paper introduces a patch-based training method for INR-based GANs that reduces computational costs and memory usage, enabling scalable high-resolution image generation without sacrificing quality.

Contribution

The authors propose a novel multi-stage patch-based training approach that decouples computational cost from image resolution in INR-GANs, improving scalability and efficiency.

Findings

Reduces GPU memory usage during training.

Maintains competitive FID scores on benchmark datasets.

Enables training of high-resolution INR-GANs efficiently.

Abstract

Recent studies have shown remarkable progress in GANs based on implicit neural representation (INR) - an MLP that produces an RGB value given its (x, y) coordinate. They represent an image as a continuous version of the underlying 2D signal instead of a 2D array of pixels, which opens new horizons for GAN applications (e.g., zero-shot super-resolution, image outpainting). However, training existing approaches require a heavy computational cost proportional to the image resolution, since they compute an MLP operation for every (x, y) coordinate. To alleviate this issue, we propose a multi-stage patch-based training, a novel and scalable approach that can train INR-based GANs with a flexible computational cost regardless of the image resolution. Specifically, our method allows to generate and discriminate by patch to learn the local details of the image and learn global structural information by a novel reconstruction loss to enable efficient GAN training. We conduct experiments on several benchmark datasets to demonstrate that our approach enhances baseline models in GPU memory while maintaining FIDs at a reasonable level.