Learning Illumination Control in Diffusion Models

TL;DR

This paper introduces an open-source pipeline for learning illumination control in diffusion models, enabling better lighting adjustments in images without relying on proprietary data or tools.

Contribution

It presents a fully open-source, reproducible method for training diffusion models to control illumination using a novel data engine and publicly available resources.

Findings

Significant improvements over baseline models in perceptual and structural similarity.

The pipeline is fully open-source and reproducible.

Achieves better identity preservation in lighting control.

Abstract

Controlling illumination in images is essential for photography and visual content creation. While closed-source models have demonstrated impressive illumination control, open-source alternatives either require heavy control inputs like depth maps or do not release their data and code. We present a fully open-source and reproducible pipeline for learning illumination control in diffusion models. Our approach builds a data engine that transforms well-lit images into supervised training triplets consisting of a poorly-illuminated input image, a natural language lighting instruction, and a well-illuminated output image. We finetune a diffusion model on this data and demonstrate significant improvements over baseline SD 1.5, SDXL, and FLUX.1-dev models in perceptual similarity, structural similarity, and identity preservation. Our work provides a reproducible solution built entirely with open-source tools and publicly available data. We release all our code, data, and model weights publicly.