MIFO: Learning and Synthesizing Multi-Instance from One Image

TL;DR

This paper introduces MIFO, a novel method for learning and synthesizing multi-instance semantics from a single image, effectively handling similar and rare instances through attention-based optimization.

Contribution

The paper presents a penalty-based attention optimization and box control techniques to disentangle and accurately synthesize multi-instance semantics from one image, a challenging task due to limited data.

Findings

Achieves disentangled, high-quality semantic synthesis

Balances editability and instance consistency effectively

Remains robust with similar or rare instances

Abstract

This paper proposes a method for precise learning and synthesizing multi-instance semantics from a single image. The difficulty of this problem lies in the limited training data, and it becomes even more challenging when the instances to be learned have similar semantics or appearance. To address this, we propose a penalty-based attention optimization to disentangle similar semantics during the learning stage. Then, in the synthesis, we introduce and optimize box control in attention layers to further mitigate semantic leakage while precisely controlling the output layout. Experimental results demonstrate that our method achieves disentangled and high-quality semantic learning and synthesis, strikingly balancing editability and instance consistency. Our method remains robust when dealing with semantically or visually similar instances or rare-seen objects. The code is publicly available at https://github.com/Kareneveve/MIFO