Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves

TL;DR

This paper introduces VisualAtoms, a synthetic dataset generated through a systematic methodology using circular harmonics, which effectively pre-trains vision transformers achieving near state-of-the-art accuracy with fewer images.

Contribution

The work develops a novel systematic approach for designing contour-oriented synthetic datasets, significantly improving pre-training effectiveness for vision transformers.

Findings

VisualAtom-21k achieves 83.7% top-1 accuracy on ImageNet-1k.

Synthetic dataset quality can continue to improve over time.

FDSL with synthetic data addresses privacy and labeling issues.

Abstract

Formula-driven supervised learning (FDSL) has been shown to be an effective method for pre-training vision transformers, where ExFractalDB-21k was shown to exceed the pre-training effect of ImageNet-21k. These studies also indicate that contours mattered more than textures when pre-training vision transformers. However, the lack of a systematic investigation as to why these contour-oriented synthetic datasets can achieve the same accuracy as real datasets leaves much room for skepticism. In the present work, we develop a novel methodology based on circular harmonics for systematically investigating the design space of contour-oriented synthetic datasets. This allows us to efficiently search the optimal range of FDSL parameters and maximize the variety of synthetic images in the dataset, which we found to be a critical factor. When the resulting new dataset VisualAtom-21k is used for pre-training ViT-Base, the top-1 accuracy reached 83.7% when fine-tuning on ImageNet-1k. This is close to the top-1 accuracy (84.2%) achieved by JFT-300M pre-training, while the number of images is 1/14. Unlike JFT-300M which is a static dataset, the quality of synthetic datasets will continue to improve, and the current work is a testament to this possibility. FDSL is also free of the common issues associated with real images, e.g. privacy/copyright issues, labeling costs/errors, and ethical biases.