Dataset Enhancement with Instance-Level Augmentations

TL;DR

This paper introduces an instance-level data augmentation technique using diffusion models to repaint objects within images, enhancing model performance and enabling data anonymization for various datasets.

Contribution

It presents a novel method for instance-level augmentation with diffusion models, surpassing traditional pixel-based transformations, applicable to multiple datasets and tasks.

Findings

Improves performance of segmentation and detection models

Enables effective data anonymization

Provides synthetic dataset expansions

Abstract

We present a method for expanding a dataset by incorporating knowledge from the wide distribution of pre-trained latent diffusion models. Data augmentations typically incorporate inductive biases about the image formation process into the training (e.g. translation, scaling, colour changes, etc.). Here, we go beyond simple pixel transformations and introduce the concept of instance-level data augmentation by repainting parts of the image at the level of object instances. The method combines a conditional diffusion model with depth and edge maps control conditioning to seamlessly repaint individual objects inside the scene, being applicable to any segmentation or detection dataset. Used as a data augmentation method, it improves the performance and generalization of the state-of-the-art salient object detection, semantic segmentation and object detection models. By redrawing all privacy-sensitive instances (people, license plates, etc.), the method is also applicable for data anonymization. We also release fully synthetic and anonymized expansions for popular datasets: COCO, Pascal VOC and DUTS.