ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets

TL;DR

ChimeraLoRA introduces a multi-head LoRA approach combining class-shared and image-specific adapters to generate diverse, detail-rich synthetic datasets that improve classification in data-scarce domains.

Contribution

The paper proposes a novel multi-head LoRA method that separates class priors and image details, enhancing synthetic data diversity and fidelity for better downstream performance.

Findings

Synthesized datasets are diverse and detail-rich.

Improved classification accuracy on downstream tasks.

Method outperforms single-head LoRA approaches.

Abstract

Beyond general recognition tasks, specialized domains and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to supplement underrepresented regions of real data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While an image-wise LoRA trained on a single image captures fine-grained details yet offers limited diversity, a class-wise LoRA trained over all shots produces diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into a class-shared LoRA~$A$ for class priors and per-image LoRAs~$\mathcal{B}$ for image-specific characteristics. To expose coherent class semantics in the shared LoRA~$A$, we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose $A$ with a mixture of $\mathcal{B}$ using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images are both diverse and detail-rich while closely aligning with the few-shot real distribution, yielding robust gains in downstream classification accuracy.