PASTA: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization

TL;DR

PASTA is a simple augmentation method that perturbs the amplitude spectra of synthetic images in the Fourier domain, especially high-frequency components, to enhance synthetic-to-real domain generalization across multiple vision tasks.

Contribution

This paper introduces PASTA, a novel Fourier-based augmentation strategy that improves out-of-the-box synthetic-to-real generalization performance for various vision tasks.

Findings

PASTA outperforms complex state-of-the-art methods in syn-to-real tasks.

Perturbing high-frequency Fourier components yields significant generalization gains.

PASTA is complementary to existing methods and effective across multiple vision benchmarks.

Abstract

Synthetic data offers the promise of cheap and bountiful training data for settings where labeled real-world data is scarce. However, models trained on synthetic data significantly underperform when evaluated on real-world data. In this paper, we propose Proportional Amplitude Spectrum Training Augmentation (PASTA), a simple and effective augmentation strategy to improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. PASTA perturbs the amplitude spectra of synthetic images in the Fourier domain to generate augmented views. Specifically, with PASTA we propose a structured perturbation strategy where high-frequency components are perturbed relatively more than the low-frequency ones. For the tasks of semantic segmentation (GTAV-to-Real), object detection (Sim10K-to-Real), and object recognition (VisDA-C Syn-to-Real), across a total of 5 syn-to-real shifts, we find that PASTA outperforms more complex state-of-the-art generalization methods while being complementary to the same.