Benchmarking and Analyzing Generative Data for Visual Recognition

TL;DR

This paper introduces GenBench, a comprehensive benchmark for evaluating generative data in visual recognition, along with a new metric CLER, and analyzes the effectiveness of generative data compared to retrieval methods.

Contribution

It presents GenBench, a large benchmark dataset, proposes CLER as a new evaluation metric, and provides insights into the advantages and limitations of generative data for visual recognition.

Findings

Generative data can be effective for visual recognition tasks.

CLER correlates better with downstream performance than existing metrics.

External knowledge injection improves recognition performance in most cases.

Abstract

Advancements in large pre-trained generative models have expanded their potential as effective data generators in visual recognition. This work delves into the impact of generative images, primarily comparing paradigms that harness external data (\ie generative \vs retrieval \vs original). Our key contributions are: \textbf{1) GenBench Construction:} We devise \textbf{GenBench}, a broad benchmark comprising 22 datasets with 2548 categories, to appraise generative data across various visual recognition tasks. \textbf{2) CLER Score:} To address the insufficient correlation of existing metrics (\eg, FID, CLIP score) with downstream recognition performance, we propose \textbf{CLER}, a training-free metric indicating generative data's efficiency for recognition tasks prior to training. \textbf{3) New Baselines:} Comparisons of generative data with retrieved data from the same external pool help to elucidate the unique traits of generative data. \textbf{4) External Knowledge Injection:} By fine-tuning special token embeddings for each category via Textual Inversion, performance improves across 17 datasets, except when dealing with low-resolution reference images. Our exhaustive benchmark and analysis spotlight generative data's promise in visual recognition, while identifying key challenges for future investigation.