Evaluating Text-to-Image Diffusion Models for Texturing Synthetic Data

TL;DR

This paper evaluates whether pretrained text-to-image diffusion models improve synthetic data quality for robotic manipulation, finding they perform similarly to simpler random textures in training models for real-world tasks.

Contribution

It compares diffusion-based texturing with random textures for synthetic data, revealing no significant advantage of diffusion models in this context.

Findings

Diffusion-based texturing performs on par with random textures.

Using diffusion models does not significantly improve model performance on real data.

Synthetic data with diffusion textures offers no clear benefit over simpler methods.

Abstract

Building generic robotic manipulation systems often requires large amounts of real-world data, which can be dificult to collect. Synthetic data generation offers a promising alternative, but limiting the sim-to-real gap requires significant engineering efforts. To reduce this engineering effort, we investigate the use of pretrained text-to-image diffusion models for texturing synthetic images and compare this approach with using random textures, a common domain randomization technique in synthetic data generation. We focus on generating object-centric representations, such as keypoints and segmentation masks, which are important for robotic manipulation and require precise annotations. We evaluate the efficacy of the texturing methods by training models on the synthetic data and measuring their performance on real-world datasets for three object categories: shoes, T-shirts, and mugs. Surprisingly, we find that texturing using a diffusion model performs on par with random textures, despite generating seemingly more realistic images. Our results suggest that, for now, using diffusion models for texturing does not benefit synthetic data generation for robotics. The code, data and trained models are available at \url{https://github.com/tlpss/diffusing-synthetic-data.git}.