NOFT: Test-Time Noise Finetune via Information Bottleneck for Highly Correlated Asset Creation

TL;DR

This paper introduces NOFT, a quick and efficient test-time noise finetuning method using an information bottleneck to generate highly correlated, diverse, and high-quality images in diffusion models.

Contribution

The paper proposes a novel plug-and-play noise finetune module employing an information bottleneck with minimal parameters and training time, enhancing image diversity and fidelity.

Findings

NOFT produces high-fidelity, diverse images with topology and texture alignment.

It requires only 14K parameters and 10 minutes of training.

Demonstrates effectiveness across 2D/3D assets with text or image guidance.

Abstract

The diffusion model has provided a strong tool for implementing text-to-image (T2I) and image-to-image (I2I) generation. Recently, topology and texture control are popular explorations, e.g., ControlNet, IP-Adapter, Ctrl-X, and DSG. These methods explicitly consider high-fidelity controllable editing based on external signals or diffusion feature manipulations. As for diversity, they directly choose different noise latents. However, the diffused noise is capable of implicitly representing the topological and textural manifold of the corresponding image. Moreover, it's an effective workbench to conduct the trade-off between content preservation and controllable variations. Previous T2I and I2I diffusion works do not explore the information within the compressed contextual latent. In this paper, we first propose a plug-and-play noise finetune NOFT module employed by Stable Diffusion to generate highly correlated and diverse images. We fine-tune seed noise or inverse noise through an optimal-transported (OT) information bottleneck (IB) with around only 14K trainable parameters and 10 minutes of training. Our test-time NOFT is good at producing high-fidelity image variations considering topology and texture alignments. Comprehensive experiments demonstrate that NOFT is a powerful general reimagine approach to efficiently fine-tune the 2D/3D AIGC assets with text or image guidance.