Stylized Structural Patterns for Improved Neural Network Pre-training

TL;DR

This paper introduces a novel synthetic data generation method using neural fractals and reverse stylization, significantly reducing the domain gap and improving model performance in vision tasks when real data is scarce.

Contribution

It presents an improved neural fractal formulation and a reverse stylization technique to create more effective synthetic datasets for neural network pre-training.

Findings

Lowered domain gap between synthetic and real images using KID.

11% reduction in FID for diffusion models trained on synthetic data.

Over 10% accuracy improvement on ImageNet-100 with ViT-S.

Abstract

Modern deep learning models in computer vision require large datasets of real images, which are difficult to curate and pose privacy and legal concerns, limiting their commercial use. Recent works suggest synthetic data as an alternative, yet models trained with it often underperform. This paper proposes a two-step approach to bridge this gap. First, we propose an improved neural fractal formulation through which we introduce a new class of synthetic data. Second, we propose reverse stylization, a technique that transfers visual features from a small, license-free set of real images onto synthetic datasets, enhancing their effectiveness. We analyze the domain gap between our synthetic datasets and real images using Kernel Inception Distance (KID) and show that our method achieves a significantly lower distributional gap compared to existing synthetic datasets. Furthermore, our experiments across different tasks demonstrate the practical impact of this reduced gap. We show that pretraining the EDM2 diffusion model on our synthetic dataset leads to an 11% reduction in FID during image generation, compared to models trained on existing synthetic datasets, and a 20% decrease in autoencoder reconstruction error, indicating improved performance in data representation. Furthermore, a ViT-S model trained for classification on this synthetic data achieves over a 10% improvement in ImageNet-100 accuracy. Our work opens up exciting possibilities for training practical models when sufficiently large real training sets are not available.