Diffusion-based Visual Anagram as Multi-task Learning

TL;DR

This paper introduces a multi-task learning framework for diffusion-based visual anagram generation, addressing concept segregation and domination issues through novel optimization and noise balancing techniques.

Contribution

It proposes a new multi-task learning approach with anti-segregation and noise balancing strategies to improve visual anagram generation using diffusion models.

Findings

Enhanced visual anagram quality demonstrated by qualitative results.

Quantitative metrics show improved concept diversity and overlap.

Method outperforms existing diffusion-based approaches in generating complex anagrams.

Abstract

Visual anagrams are images that change appearance upon transformation, like flipping or rotation. With the advent of diffusion models, generating such optical illusions can be achieved by averaging noise across multiple views during the reverse denoising process. However, we observe two critical failure modes in this approach: (i) concept segregation, where concepts in different views are independently generated, which can not be considered a true anagram, and (ii) concept domination, where certain concepts overpower others. In this work, we cast the visual anagram generation problem in a multi-task learning setting, where different viewpoint prompts are analogous to different tasks,and derive denoising trajectories that align well across tasks simultaneously. At the core of our designed framework are two newly introduced techniques, where (i) an anti-segregation optimization strategy that promotes overlap in cross-attention maps between different concepts, and (ii) a noise vector balancing method that adaptively adjusts the influence of different tasks. Additionally, we observe that directly averaging noise predictions yields suboptimal performance because statistical properties may not be preserved, prompting us to derive a noise variance rectification method. Extensive qualitative and quantitative experiments demonstrate our method's superior ability to generate visual anagrams spanning diverse concepts.