Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection

TL;DR

This paper introduces HEaD, a method for early detection of hallucinations in diffusion models, which reduces computational waste and speeds up image generation by predicting incorrect outputs early in the process.

Contribution

The paper presents HEaD, a novel early detection paradigm that combines cross-attention maps and a new indicator to forecast final outputs in diffusion models.

Findings

HEaD reduces generation time by up to 12% in two-object scenarios.

Early detection with HEaD improves efficiency and accuracy in complex image generation.

HEaD demonstrates the importance of early intervention in generative AI processes.

Abstract

Diffusion models have significantly advanced generative AI, but they encounter difficulties when generating complex combinations of multiple objects. As the final result heavily depends on the initial seed, accurately ensuring the desired output can require multiple iterations of the generation process. This repetition not only leads to a waste of time but also increases energy consumption, echoing the challenges of efficiency and accuracy in complex generative tasks. To tackle this issue, we introduce HEaD (Hallucination Early Detection), a new paradigm designed to swiftly detect incorrect generations at the beginning of the diffusion process. The HEaD pipeline combines cross-attention maps with a new indicator, the Predicted Final Image, to forecast the final outcome by leveraging the information available at early stages of the generation process. We demonstrate that using HEaD saves computational resources and accelerates the generation process to get a complete image, i.e. an image where all requested objects are accurately depicted. Our findings reveal that HEaD can save up to 12% of the generation time on a two objects scenario and underscore the importance of early detection mechanisms in generative models.